TOPIC:

For several years, our club has been printing weekly agenda in our own format (no one likes the standard agenda format very much and there appears to be no other option for formatting it otherwise.). It entails a lot of copying and pasting and can take several hours each meeting to prepare an agenda. I started formatting a screen-scraper to collect the various parts that we need and export them to a CSV formatted file, then read them into a document which is better, but still has issues because of the way some of the sections are coded. . It involves having to parse the contents with REGEX. It would have to be redone if and when there is a re-design of or change to the rostertable.
It occurs to me that there are several issues with the website that could be solved if there were an API (or a download option) where we could retrieve a chunk of data (A JSON object or an XML file) with the individual fields separated and identified.

One of the other issues it could potentially solve is the issue of identifying what roles former members performed. Currently, if we look at older agenda through the website, the role is simply designated as "former member". Like many clubs, we occasionally have members who leave, then rejoin, or suspend membership for a period due to various reasons. Reconstructing their history is just more work that could be avoided if the data were complete. We could use data from each week's meeting to update our own web-centric database, format our own agenda, and better track member progress.

I'd also be very interested in a JSON feed for exactly the same reason.


empulse: Could you share your scraper code?

It might be worth, if you aren't already, parsing this page instead as it appears to be far cleaner:

/index.cgi?printagendasignup

I'd also like to see JSON feeds for agendas for the same reason.

Thanks,

Ian

Adding my +1 to this. Our club uses an Excel-based agenda maker that a former member put together some years ago. It works well enough for folks who have Excel, but requires manual input of most agenda items. I'd love to be able to build a platform-agnostic tool to handle this task.

Still working on my scraper code. There are issues in the way that the agenda is coded on the website that make scraping the site extremely difficult.

Regarding printing custom formatted agenda*: I have partially (and probably temporarily) solved the issue for our club by creating a template with Libre Office (also works in Open Office) that makes prepping our agenda much quicker, it does, however, still require some cutting and pasting. The template may work with Microsoft word processing as well but II don't use that software.

Because each club has their own format and often uses slightly different terminology, An API or some other way to access the data stored for agenda is the most flexible option.

I am attaching a zip of my template for anyone who might want to use it as an example. Also attaching a zip of the final PDF.
It is a two column document with a table in each column. The borders serve as a guide when prepping the document but I switch the borders on the right side table off before printing

FYI: Open Office and Libre Office are similar, free, open source and fully-featured office suites. Both will run on MAC, Windows or Linux systems

*(The word "agenda" is the Latin plural of agendum)

I add my plus 1 to this. I have laboriously written code to automate the process; it has been an enormous and moving target venture. Given the amount of interest as expressed in this forum can we make this a priority feature request?

I have been lax in pursuing the status that would give me access to the FTH innards I have also seen the possibility of contributing to creating this feature. What would my next step be toward gaining developer / maintainer status?

+1

I'm curious, when you are scraping the agenda page, do you authenticate to the club site within your script or are you grabbing from some secondary source, such as generated by the "print" button?

I'm trying to scrape the html using an http/get command with and without basic auth and not finding the expected fields in the resulting data. I assume this is related to the above-mentioned difficulties that result from how the page/system is coded.

Are there any helpful resources you've found that might get me over this hump?

My best : )

You need to be logged in to see the Agenda.

Thank you, Brian. I'm attempting to authenticate programmatically using an http library with the typicalcommand. I've tried passing my credentials using basic auth and I see what looks looks a portion of the correct html source, but I don't see the text "Presiding" as in Presiding Officer, which I can clearly see when I'm looking at the page or its source in my browser.

If anyone has had any luck scraping the page over http, any tips are welcome. I'll report my progress soon.

Sorry we do not provide that kind of information.

I completely understand the need to support only the core functionality of the site, Brian. I'm grateful for the support work you all do to make this platform so effective for so many.

As a work around, I'll attempt to write my first Chrome extension. My assumption is that the extension should have access to the full content of the agenda when I'm logged in, unlike my shell based script. I'll test this assumption and report my findings. My best : )