F@H Stat Banners

[epicbard]

Originally Posted by Protoplasm72

No need. I dig through uncommented lines of code every day.

allright... what is it you do?

it should be noted that i am a pretty crappy coder... ><

oh i am also sending you my original perl script that i no longer use. you might find it handy i dunno. i switched to python bc PIL was that much better than perls Image Magic

[k0NG0]

Originally Posted by Protoplasm72

No need. I dig through uncommented lines of code every day.

Do you suffer from red eyes and all that?

Python > Perl, that's what I've started to feel lately. There's one downside to vBulletin and that's that it's in PHP. Me no likey PHP.

Just wanted to say good job on the banners. Hopefully I can get the ball rolling on the F@H extension to vB that I've had in the back of my mind for over 6 months now. Feel like I have enough focus to pull that off. That could mean that we could integrate the banner with our own database instead of pulling it from the F@H servers. Anyway, I'm sketching out a database schema and stuff. Could be quite nice.

[Protoplasm72]

What did you have in mind kONGO? I wrote this before epicbard sent me his script (which is in perl). It uses a mysql db to store the stats. If a requested user is not in the db yet or the stored stats are more then 3 hours old it retrieves them from the EOC stats and loads them into the db. So basically it retrieves the stats on demand and caches them to prevent hitting the EOC site everytime an image is loaded. I just have to figure out what is going on with the url encoding that causes names like [AB]Team_AthlonPower to fail

http://thelonehacker.com/folding/fol...e=protoplasm72


I'm probably going to convert epicbard's image creating script to php. If I had a site that ran perl, php, and had a mysql db I might just try hook the 2 together but that wouldn't be a pretty design .


By the way this is my first time using php and I'm not a big fan yet either. I'm mainly a java programmer so it seems really unorganized to me.

[k0NG0]

PHP is bug city. Really, I think it's among the highest ranked in # of bugs for open source projects. It's easy enough to learn and with 5.x the object model stopped sucking horribly, which was a good thing. Still, I don't like it for some reason. Of course, web applications tend to be quite horrible either way, since MVC usually means you still have to have some sort of pseudo code in the view part's template facilities, which goes against the grain of the whole separation, but still.

My idea was basically to mirror all the stats on-site for the top 1000 teams (and their members) or something. Having a daemon-type system (prolly just a cron-ed script, no overkill needed, even though I've thought about having a static daemon process and calling via DBUS or just shared memory) that yanks the stats off the F@H servers and pushes it into an on-side database. Then vB integration with stats directly available on a user's profile page and a dedicated "front page" for the vB team, with integration of the distributed computing discussions and such. Graphs and everything could be run off the on-site data, which is faster, and by caching the generated images according to some sane algorithm it's basically pseudo-static data, which means there isn't a need to let PHP run every single time you generate an image or a graph.

(A bit like how Blogger runs, it's basically a Java-based front-end that generates and pushes static HTML via FTP to the web servers for serving--something I've deduced from various error messages Blogger has thrown at me.)

[DeusExMachina]

Problem with every stat is that there will be a limit of how many times to access the original stats (from stanford or EOC's page).

What would be nice is an ability to export the stats for the team and/or individual to their own website, and use their own bandwidth/database. No resource limits. Not everyone has their own webspace, for those of us who do, it'll help ease the strain on resources (and we can use our own sig designs).

May I suggest an db installer proggie (like some of the advanced vB mods have)? With a how-to to configure the variables (like which fields to display)? That would eliminate the main bottleneck of a central stats server (if it goes down for any reason, bye bye stats and even image hosting), meanwhile offering customizing options tweakers crave.

DXM

[k0NG0]

Is there really a limit? Grabbing the .txt files is the nicest way of doing it, and if the servers use entity tags, one only has to grab 'em when they're really updated (the way it's meant to be done). I'm not aware of a limit per-IP placed on grabbing those files through regular HTTP (of course, I haven't researched that part very much).

Those .txt files are the closest thing to a clean stats export from the F@H servers. Grabbing the HTML versions and scraping them is too much work and, frankly, dumb.

[DeusExMachina]

Usually there's a limit on the scraping. And the EOC site has a warning of banning IPs that grab the entire HTTP.

They have XML. If someone can do an XML (XSLT version be nice for those with webspace with Saxon and such installed), it can be distributed anywhere -- on websites/blogs/sigs you name it.

After surveying the competition (especially p2p-community.com), we need to get our folding teams and promotion better organized. All the top teams got their production going assembly style, which seems to attract the most crunchers -- we have a lot signed up, but too many aren't folding.

Better sigs, more visual stats and more rah rah will help!

After these 2 WUs are finished, I'm going to work on a sig image that would rival all the rest -- eye popping beautiful, and hopefully under 10kb (crossing fingers). Just would need someone to do the coding for the stats and do the positioning (have worked with such php sig scripts before for things like server stats, but not this).

DXM

[k0NG0]

The .txt is like an XML data file, just without the added fluff. For these stats, the tab-separated data file is ideal, no processing or transmission overhead to speak of, which XML adds. I'm actually happy it's a fixed .txt-file format and not XML-based, as XML is a beautiful concept, but usually used in ways that are way (waaaaay!) overkill.

All I really had in mind was to grab the main team stats file, push that data into a table. Then grab the top 1000 teams' stats files and push that into another table. Doing that only when Stanford has updated the files (thanks to some E-Tag magic, if the servers are hooked up to support that, which they should since, well, it's bloody Stanford!). That way you just grab data when it's fresh and you store it on-site which means all actual front-end code only touches the local copy, the back-end handles all the database updating.

But you're right we need more focus on our F@H team. That's what I hope we could achieve with tight integration between the F@H team and the forums at large. Possibly create some nifty pages that other folders would like which could bring in more ad revenues too... Statistical analysis and such.

[DeusExMachina]

Why pull the stats for all the top 1000 teams? For comparison purposes? That seems like a lot of work, when it's just ABX.com stats that's needed for sigs.

There are ready made php sig scripts for vB at vbulletin.org for ideas. Already pre-configured for vB, and with a little additional coding, can pull the variables from the new tables right out of vB's db. But strongly suggest to put it in it's own db, as vB's db is large enough as it is!!

DXM

[k0NG0]

Yup, for comparison purposes. Movers 'n' shakers, that sort of thing. Perhaps launching a F@H stats service that rocks the hardest (Stanford's ain't exactly hot). To attract users and thereby attracting more advertisement revenue by showing "here's this page, you could advertise this here and here's how many would see it".

For sigs it's dead easy, yes. I have such a PHP script already, that yanks the stats in real-time off the Stanford servers and adds it to an image. Less than 100 LOC, IIRC. Wrote it over a year ago.

[Protoplasm72]

Sounds like you want something like EOC's stat page. There are many forums that do this but they are the best I've seen. It requires a lot of resources though and more then a couple hours to code it. http://folding.extremeoverclocking.com/?nav=DBINFO

[k0NG0]

Ya, I know it'll be a big system. Still, that's sort of what I've had in the back of my mind. Perhaps on a somewhat lesser scale. I dunno, I've only been tossing around ideas, nothing is written in stone (or in code, for that matter) yet.

[Protoplasm72]

My next project I'm planning on doing is to make a better online monitoring program for all my folding boxes. Notice the 'My folding details' link in my sig only contains the machines on my home network. I want to setup a db on my website that I can post my personal stats to from anywhere and then generate protein details, stats, graphs, etc.. The online part with the db and php scripts should be easy. I still have a little work planning how I'm going to get machines to post my stats from windows and linux without having to write my own monitoring tools.


I've looked around and there aren't any tools like this. There are good monitoring tools for windows and linux that can post results to a website like the link in my sig but nothing that combines the results from various networks and os's. I think this might actually be a good tool for abxzone to offer its folders. Give people on our team a place to post their personal detailed stats. It might draw in some folding enthusiasts that want a better way to track their machines that are in remote locations.

[k0NG0]

Sounds like a great idea. That could be worked into the forums too, yes. It would be neat to have deep F@H integration like that. Are you thinking about writing some sort of cross-platform wrapper for the various programs? I've thought about writing something with XPCOM and the Mozilla stuff, but I fear it would be overkill.

I've wondered how EMIII and those tools yank the data out of the F@H process, but I haven't looked into it with any depth at all. I had something in mind along those lines too, a more cross-platform, probably script-based, library for F@H stats. Uniform API across all platforms supported by F@H.

[Protoplasm72]

You're more ambitious then me. My first goal is to get the data to my db in the simplest way possible. That means use the csv generated by em3 and post the new values to the db on my webserver. Probably with a perl script or something. I don't have a monitoring tool on my linux box yet but I was hoping to find one that generates a similar csv file with the completed protein data.

Once I get a simple version running I'd try to write something more like what you described. I was thinking a simple java app to parse the fahlog.txt (this is the main way for monitoring programs to get their data... em3 also uses current.xyz and other files in the work folder to generate the drawing of the proteins and better estimates of current progress within a frame). I was thinking java because I know that best but it admittedly is not the most efficient way.

It all depends on how lazy I'm feeling . I've got a couple projects that I started and never finished because new more exciting things keep coming up.