lxml is a bit of a mindtwister and I only know as much as I need to as more advanced things require XPath. If you’re trying to get your head around how all this works, I suggest taking a look at my other two scrapers which are considerably simpler:
As I learn more I take on more challenging tasks which leads to more complex scrapers, but if you know java and regex, python should be a breeze. I don’t mind answering specific questions or talking on skype if you want to go through the code live. Duplication of effort is a pet peeve of mine, and using scraperwiki/python/lxml has been like having a new superpower I’d love to share. Don’t hesitate to ask if you’re willing to invest some time.
Don’t hesitate to ask if you’re willing to invest some time.
Deal! I’ll read some about this and look into it more. I’m interested in this in that it seems like it’s somehow… welll… “scraping” without digging through the actual html? Or is that not right? I have to all kinds of dumb stuff to the raw html, where as this seems like you’re able to just tell it, “Get td[0] and store it as =variable= for all the tr’s.”
It’s pretty slick. But… maybe the method itself is actually digging through the raw html and collecting stuff that way. Not sure.
Yeah, lxml processes all the html into a tree and gives you an API so you can access it as you like. It takes a lot of the grunt work out of extracting data from HTML.
Which is awesome, as I just felt the pain of hand pruning a heckuva lot of html tags out of something I wanted to transform to a different format. Even with my find-replacing, line breaks would prevent the tag from getting detected fully and I had to do a lot of tedious stuff :)
lxml is a bit of a mindtwister and I only know as much as I need to as more advanced things require XPath. If you’re trying to get your head around how all this works, I suggest taking a look at my other two scrapers which are considerably simpler:
http://scraperwiki.com/scrapers/commonsenseatheism_-_atheism_vs_theism_debates/ http://scraperwiki.com/scrapers/hp_fanfic_reviews/
As I learn more I take on more challenging tasks which leads to more complex scrapers, but if you know java and regex, python should be a breeze. I don’t mind answering specific questions or talking on skype if you want to go through the code live. Duplication of effort is a pet peeve of mine, and using scraperwiki/python/lxml has been like having a new superpower I’d love to share. Don’t hesitate to ask if you’re willing to invest some time.
Deal! I’ll read some about this and look into it more. I’m interested in this in that it seems like it’s somehow… welll… “scraping” without digging through the actual html? Or is that not right? I have to all kinds of dumb stuff to the raw html, where as this seems like you’re able to just tell it, “Get td[0] and store it as =variable= for all the tr’s.”
It’s pretty slick. But… maybe the method itself is actually digging through the raw html and collecting stuff that way. Not sure.
Yeah, lxml processes all the html into a tree and gives you an API so you can access it as you like. It takes a lot of the grunt work out of extracting data from HTML.
Which is awesome, as I just felt the pain of hand pruning a heckuva lot of html tags out of something I wanted to transform to a different format. Even with my find-replacing, line breaks would prevent the tag from getting detected fully and I had to do a lot of tedious stuff :)