Edit: The Lilina website has moved here. I believe this new incarnation has incorporated many of my hacks, so I would recommend checking it out.
Lilina-head is my hack of the lilina aggregator. This version is a big improvement on my original patch file, which was furiously coded up in just a few hours. Be aware that the default style is drastically different from the original lilina. I include the original lilina style if you prefer it (see the conf.php file), but be aware that it might require a little bit of twiddling.
Changes I've made to the original:
- added categories.
- single source page. clicking on a source's icon will take you to a page displaying only headlines from that source.
- include your delicious links after the headlines (any delicious links with a tag matching the current category will be included).
- added paging (so headlines can be distributed across several pages if necessary). the default is 100 headlines per page.
- added a max number of headlines option. i have close to a 100 sources, so if i hit show all i will sometimes get over 1000 headlines (which is completely ridiculous even if they are paged). there is also the risk this many headlines will cause lilina to exceed the memory limits allocated for a php application. to prevent all this i added the ability to set an upper limit on the total number of headlines. default is 1000.
- sort of similar, i added a max headlines per source configuration option. no more headlines than this will be taken from any one source. default is 20.
- ability to remove images from feed excerpts (this can really speed things up).
- changed the way lilina creates icons for the sources. lilina-head will try and grab a favicon if it can (which generally look better, imho), and then it will try for an image from the source feed, which it then randomly zooms and slices into the appropriate size. the reason for the zooming and slicing is that there is apparently no concensus on the size and shape of feed images.
- all icons are now cached, instead of being served up by the source website. the downside to this is if the website changes their favicons/images it won't be reflected on your aggregator. to compensate, i added a fetch icon link in the admin panel so you can re-fetch the sites icon.
- added the ability to upload an image for use as an icon for any source. use this if a site doesn't have an image, or if you just don't like it.
- customizable default icon if all else fails.
Things I need to do:
- when uploading an icon, not all methods are implemented. right now only zoom works.
- the show feeds time settings should be customizable. that is, someone with many sources may want these to be 3h/8h/24h/72h, whereas someone with only a few sources (infrequently updated) may want them to be 24h/48h/72h/week. it would be nice if this could be set in the configuration file.
Please take a look at the configuration file. There are a couple of new settings there.
NOTE: It's been brought to my attention the icon fetching routine bombs out if you don't have php >= 4.3. Sorry about that. I'll see what I can do.
Hello,
I'm really interested in your hack because I want to implement a planet with it. Contact me if you need some help, nice work ! Waiting impatiently for v0.2 ;)
Very nice indeed. Many thanks for making the patch available. Had it up and running without too much of a problem.
Thought you might want to know that one of the image files missed being packaged in the zip file. The image is 'small_blue_dot' under the 'img' package.
Also, would know why the loading of feeds is excruciatingly slow sometimes?
Thanks Aregon. I will re-zip.
As for why it is really slow sometimes, I think it is because of the way the Magpie cache works (Magpie is the RSS backend).
Say you have 20 feeds and the Magpie cache set to 1 hour. If you don't visit your site a lot, when you do visit it is quite possible many of the feeds cache have expired (say maybe 1/2 of them). Regardless, Magpie will now have to fetch the new feeds. If you visit again in another 20 minutes, maybe another 5 feeds cache have expired. Once again Magpie has to fetch new feeds, which causes things to slow down.
The way to avoid all this is to essentially disable the Magpie cache all together, and schedule an hourly process to update all the feeds for you. To do this set the $CACHE_IN_HOURS = 168 (any time frame less frequent than your scheduled updates should do) in the conf.php file. Now you will need to set up a cron job (or, on windows you can use Task Scheduler) to hourly fetch the page, being sure to pass force_update=1 in the query string.
Using wget/crontab on linux this would look like this:
15 * * * * /usr/bin/wget -nc -q -O - http://your_website/index.php?force_update=1 >/dev/null
This will refresh all your feeds 15 minutes after the hour.
I think wget might be available for windows too (certainly under cygwin). If not there should be some other command line web fetching app. The important part is the force_update=1 in the query string. This causes lilina to ignore the cache. Essentially this causes all feeds to be updated once an hour. At any other time these "stale" feeds are returned, which causes things to be faster and is precisely what you want.
Note: Right after adding any new feeds, it will be slower as Magpie has to fetch these feeds for the first time. This is unavoidable.
I'm trying to set this up on my site, after I had success with my previous lilina install. I'm a huge fan of the changes you've made, and really want to use them. My problem is that sites that don't put a time stamp on their posts get a stamp at the time that they were fetched, but do not appear in the default feeds page. They only show up when "all" feeds are selected. See Fark for an example on my install.
Any workarounds, or is this a known bug/feature?
Mike,
my guess is that there is a problem with the time.data file. This is where the time of retrieval for feeds that don't carry timestamps is kept. Check in the conf.php file to see where this file is (default is time.data), then check to make sure this file exists and is being updated (look at the time it was last updated to tell). If it doesn't exist create it (touch time.data or else just upload an empty file). If it's not being updated it is most likely a permissions issue. lmk if that fixes the problem.
All of that was working, and set up properly as far as I can tell. The times ARE getting updated, and the stamps are set properly. The items just don't show up in any of the time based views. When they are displayed (by clicking the "All" link or by going to the specific feed's page itself, the time stamps are all more or less right (The time that they were retrieved initially). They just don't show up in the main view.
I'm going to hack around with it a bit more tonight, to see what I can figure out. If it's not a known issue, then I'm sure it's just something stupid that I've done to my copy. I'll report back with my findings, either way.
Got it, I think. I need to let it run for a few hours to see for sure, but preliminary testing seems to have found the problem.
Lines 75 and 76 of index.php
if ($x['date_timestamp'] == '')
$x['date_timestamp'] = create_time($x['title'] . $x['link']);
Only get executed if the timestamp exists, unless the TIMERANGE > -1. So Timestamps only get applied to unstamped items when showing all items.
If you move those two lines to just above the If Timerange > -1 line (line 70), it seems to work.
I'm just not entirely sure that it will save those timestamps, although it seems that it does.
Good job Mike ! Your absolutely correct, those lines should be above that if statement, so headlines with no timestamps will be given one before that test. I appreciate it.
That if statement is something I added on, which serves the purpose of keeping the size of the headlines array down. In the original he just throws all headlines into that array (even if they are older than what is requested), then tosses them later.
One other question. I'm on the east coast, and my webhost is in California. Is there any way to add an offset to the timestamps easily?
Hey Mike,
I havn't tried it, but I think I can set you in the right direction. Find the following lines in index.php.
if ($x['date_timestamp'] == '')
$x['date_timestamp'] = create_time($x['title'] . $x['link']);
You will want to add
else $x['date_timestamp'] += 4;
or something along those lines.
Also, open up lib.php and find the function create_time. Change the line
if ($time_table[$md5] <= 0 $time_table[$md5] = time();
to
if ($time_table[$md5] <= 0 $time_table[$md5] = time() + 4;
Thanks for all the great tips. The cron job I was running wasn't nearly as clean as yours (forgot about the /dev/null dump, and was just filling up my admin account -- duh...), so thanks for that. I'm running a version of lilina at bendblogs.com and will be playing with it a bit, using some of your tips. Now I just need to make a duplicate of the index.php file and make it dump an RSS feed from all the feeds.
Again, thanks!
Hi e-head,
I've got lilina-head up and running at blogtimore.com, and it's working really well. I stumbled across it completely by accident but it ended up being exactly the solution we needed.
I'm having a bit of a problem though, as my list grows. Right now, I've got 54 sources spread out among 27 different categories. This may be pushing the limits of what it was designed to do, I don't know. We anticipate a lot more sources joining now that the site's up and running, though.
The problem is with managing the list. At times, it appears that sources are simply vanishing. Not just failing to show up on the page, but actually disappearing from lilina.data. I'm the only one administering the site, so I know it's not being done by anyone else.
Have you ever had a similar problem? I'm now keeping a backup of the lilina.data file offline, but I'm not sure what else to do.
At the moment, I'm having a problem with edit.php failing to function, too. Attempting to add or delete a category or source results in an "internal server error" after several minutes of inactivity.
This may all just be me. If I can offer any more information that can help you troubleshoot, I'd be happy to help.
Cheers, and thanks again for this cool little mod!
Oh, it was all the fault of an errant feed. Lilina was waiting 30 seconds each time the page loaded on a server that had gone down. Crap.
Plans to incorporate these modifications into the original Lilina? Or will there be a fork? Perhaps the Lilina project could expand and include e-head.
Suggestions to include 2+ (but not all) categories on one page? Thanks
Thanks for the time hint. It works, although the time is in seconds, not hours.
Mike, can you please tell me what exactly did you do to get the time moved? My server is -6 hours from my timezone, so when a newsitem is added at 19:00 local time, my site shows it at 13:00.
This is what I did and it didn't work:
On index.php page I added this (bolded):
$x['source'] = $data['feeds'][$i]['url'] ;
if ($x['date_timestamp'] == '')
$x['date_timestamp'] = create_time($x['title'] . $x['link']);
else $x['date_timestamp'] += 6;
$items[] = $x ;
On the lib.php page I added this (bolded):
function create_time($s) {
global $time_table;
$md5 = md5($s);
if ($time_table[$md5] + 6;
return $time_table[$md5];
For some reason - there's absolutely no change in the way my page is displayed. Any help would be appreciated. And e-head - absolutely awesome lilina hack! Thanks!
While I'm at it, can you guys tell me what setting do I possibly have turned off in my firefox, because when I click on individual news item even in e-head's "bored?", it doesn't expand. In fact it doesn't do anything - no error message, nothing. If I open that page in IE, it expands the news item as I click on it. So there must be something that I have customized on my firefox in the past that doesn't allow me to be able to expand single newsitems on lilina... thx!
Oh and last thing (really!) - i added an rss feed that was missing a favicon.ico. I got this error:
Fatal error: Call to undefined function: mime_content_type() in /homepages/25/d87412552/htdocs/wsc87412561/lilina2/lib.php on line 116
Then I tried uploading a favicon.ico just myself - and got the same error. Any clues?
[...] ; Administrator @ 12:17 pm
I’ve spent entirely too much time of late hacking on e-head’s lilina-head hack of lilina (an rss aggreg [...]
Yeah, I'll post what I did later on tonight. It's really pretty simple (two or 3 lines), but I don't have my changes in front of me at the moment.
Looking at what you've posted there, though, I'd suggest changing the 6's to 6 * 3600, 21600. Like I said in my post above, the time is in seconds, not hours.
[...] ed a version number for this thing, but here is the updated lilina-head. Changes from the original lilina-head include: added support for copyright, dc:rights, and creativeCommons [...]
links for 2005-06-20
lilina news aggregator (tags: del.icio.us PHP RSS Software Weblog) Dots and Loops ยป Lilina-head (tags: PHP RSS Software aggregator) Depraved Librarian: Proust Fest 2005: Swann's Song (tags: Literature NYC) The New York Times > New York Region > Inte...
Okay so the two trackbacks from Stage Directions above are me. To sum up, I've fixed a few things, updated and changed a few things, and added some things to lilina-head and the latest info on it is available at: http://www.mthespian.com/wp1.5/archives/category/lilina/
(there's a pretty big summary of changes, so I'm not going to post them all here)
Jaro, your mime_content_type error most likely indicates your php version is too old. e-head notes above that you need at least 4.3 to make it work.
O.K., sorry for the delay in responding.
Tracy S Fitch:
Thanks for helping to push things forward while I've been sleeping !
Jaro:
The expand/contract is just simple javascript. That should be the only thing you need enabled.
BH:
Did you get all your problems figured out ? Some of what you describe sounds like IO problems (maybe file permissions ?).
Ahh ! I bet I know what it is. I did notice a delay sometimes when Lilina was attempting to build the opml files for all the categories. In fact, it got so annoying I ended up changing this in my (personal) version. I've yet to figure out what the problem is ... though, in your case, with 27 categories, 27 files are being written. I should probably change this so that the opml files are only updated when the admin requests them to be updated.
Plans for the Future:
I'm furiously embroiled in this Linux project right now, but as soon as I am finished I plan to look over Fitch's code, fix one or two little bugs that have been brought to my attention, and start working on this again. Did everybody catch the bug with the timestamps and some headlines being dropped ? That's a pretty important one !
For the next version, I'd like to shove all the lilina specific code into a seperate file and have it callable via functions. This will allow index.php to be more like a template that can be more easily modified and themed, very much in the fashion of wordpress. Hopefully I can get to this within a week or two.
I fixed/tweaked/updated a few more things:
http://www.mthespian.com/wp1.5/archives/2005/06/25/lilina-head-03a/
Dammit, I totally forgot about posting my time hints, and I still don't have my code on this computer. I'll get it here tomorrow night, at the latest, and if I don't, well, you can't really do much. But believe me, I will.
Greate hack, indeed.
When I came across lilina for the first time, I thought it shuold have categories, and I implemented one, but yours is far more excellent:)
I also made an XUL page for lilina, and are you interested in making another hack? :P
all the best~
it's a nice work, i'm very glad to have a try.
there's no small_blue_dot.png in the latest zip package. maybe i could see the update soon. :)
btw, in "index.php"
maybe
or maybe there's a path error when the BASEURL is a multistep path.
maybe sth. mistake. i mean index.php line 274 about the path of a image. take "img/small_blue_dot.jpg" instead of "/img/small_blue_dot.jpg" maybe better.
BunnyQ if memory serves me correctly in e-head's original archive the small_blue_dot.png got missed. I did pull it into my updated archive that I linked to above. (and fixed the if statement placement that Mike pointed out and e-head endorsed) It's not e-head's official package I know but until he gets time for his rewrite -- perhaps it could help?
Thanks for your help and I've got the right package from the URL you mentioned above. :)
There seems to be a problem if two rss items are created within the same minute. The ID of the div will then be the some for both items. Resulting that the expanding/collapsing of the second item will expand/collaps the first item
Pascal, I suggest you check my updated file as well. I had noted the problem you describe with multiple items with the same link attribute. I fixed that in my last update to add date and time to link information in generating the item id. You can find my latest version at http://www.mthespian.com/wp1.5/archives/category/lilina .
As above, I acknowledge that this isn't e-head's release, I've just added, tweaked, and bug-fixed his project a little bit. Hopefully it can suffice until he has time for his rewrite.
Tracy: Same thing with your distro. You could check it out here: http://planet.kulturwerk118.ch/ (entries on june 27th 14:39). The RSS file seems to be good.
Another problem I just figured out: I've added one Postnuke and one Xaraya RSS file to my Lilina installation. From neither of them Lilina seems to get the date correctly (They are added as by Now instead of the date listed in the RSS file)
right you are. in essence what you have is two items posted at exactly the same day/time with exactly the same link. truthfully, maybe there isn't a reason to generate the item_id the way we do anyway. It might be just as effective to generate it via mt_rand(100000000000000000000000000000000,999999999999999999999999999999999); . (note that I haven't tried this yet so I don't know for sure. I only went to 33 digits because that's what the existing numbers do.)
Further, honestly is there any reason the item_id can't just be generated sequentially by adding one to the previous one? I'll have to play with that later.
I hope you've got that stylesheet in a separate file. I really like it and I'd love it drop it in as another option.
They're not posted *exactly* at the same time. They differ for a few seconds. Therefor if it is possible to also use the seconds portion of the date to generate the ID this will perhaps solve my problem.
The stylesheet is style_fancy.css from your archive ;-)
thank you for this. you have saved me from a life of hell and blogrolling :)
E-head: Yep, I got it! It was an errant feed... Lilina was trying to reach the server, which wasn't responding, every time the page loaded. When I took it out all was well.
Thanks to everyone who's been working on this...it's really a hot little piece of software, especially with these additions.
Hello everybody,
what a great hack. Thanks alot.
...but sorry, there is still something that I'm missing:
how can I put one feed in two categories. If I put the feed twice in the system and gave both different categories, than I have doubled feeds in the all-view. Is there any easy way to add many categories to one feed?
Karl, you don't have to put it into the system twice, you want to put the feed in one time and then check the boxes in the category view for all of the categories you want to put it in and update them. Basically a single feed record can be in multiple categories.
Strange, my host is said to be running PHP 4.3.11, yet I still get the mime_content_type error.
I'm going to completely re-evaluate all that image fetching code for the next version. Ideally, it will be compatible with php versions going pretty far back.
I also get the "mime_content_type" errror message while running PHP Version 4.3.11. Got it while trying to add this:
http://feeds.feedburner.com/maccast
I'm getting the mimetype error as well using PHP Version: 4.3.10 (Zend: 1.3.0).
Fatal error: Call to undefined function: mime_content_type() in /lib.php on line 133
It happens every time I upload a favicon. The only icons that show up are the ones that came in the tarball. Any ideas what's causing this? Also, when I hit the "fetch" link nothing happens...
Great hack by the way...
Ajax-ed Lilina
When i talk about my project list long time ago, boringest mention about doing ajax on lilina-head .
Since i was playing a bit on xajax that time , i actually testing to apply the ajax on lilina-head , at first i just testing to add a feed by ajax , ...
[...] oder man nutzt einen gehosteten Dienst. Zum selber basteln u.a.: FeedOnFeeds, Lilina 1, 2, 3 und 4, Magpierss und zFeeder. Auch via RssParser kann man Feeds in seine Webseite einba [...]
How do I correct an issue of showing the images from the feed? I get an "Image removed" where the original blog image should appear?
Also, the link to the bug fixes at
http://www.mthespian.com/wp1.5/archives/category/lilina
seems to be broken
Hi all
I have the same Problem as Jaro and Meta4 mentioned:
making icon for [feedurl]
Fatal error: Call to undefined function: mime_content_type() in /lib.php on line 133
Any ideas in this case? Its not with every feeds, most of them work very well. but with some of them, the script is not able to create a icon... Would be very apreciated to get rid of this minor bug. thanks in advance. sandro
Guys,
I must be really lame (or really tired). I set up an experimental site to pick up the RSS feeds from www.mactech.com ... and I cannot seem to add any feeds! The feed does validate, but in the end, no new news shows up.
I set the username/password for http://www2.mactech.com/edit.php to "user" and "pass" just in case someone wanted to check it out (it's not a production site, just something to experiment with).
Any ideas as to what I'm doing wrong?
If we can get this all to work in some cool way, we're going to roll it out to the tech community for Mac geeks.
Thanks,
Neil Ticktin
Publisher, MacTech Magazine
Although I haven't added many of these features yet, I'd love to. I'd also love to incorporate some of your ideas from Dots and Loops, as they look pretty interesting.
Is there anyway I can get you to generate a diff for the current version, available from Subversion?
Thanks,
Ryan.