403 errors at low record limits

[dmr5090]

Hey all - did anything change with the record limits?

I'm suddenly encountering 403 errors at what I thought would be pretty strict parameters.

After some testing, it appears the limit is 1k records. (Did it used to be 10k?)

For example, when I test with:
city = 'San Antonio'
state = 'TX'
past_days = 26
radius = 1
listing_type='sold'

I get a 403 error. But when I change past_days to 25, 976 records are returned (as of today, 2/1/25).

1k records is fairly restrictive, though I understand this may be beyond HomeHarvest's control. Is there a built-in pagination mechanism?

[alphasynth]

I've typically received 403 errors if I try to run too many requests within a given time period. I wonder if Realtor.com is rate limiting your IP address? I was able to make a request to show sold properties within the past 26. days.

On Redfin, 1,020 homes show up as sold within the past 1 month as of 2/3/2025 which appears to be consistent with the results you are receiving of ~976 homes as of 2/1/2025.

https://www.redfin.com/city/16657/TX/San-Antonio/filter/include=sold-1mo

[dmr5090]

Definitely could be too many requests, but I suspect it had to do with some rate limit on records returned. The issue was when I switched past_days to 26 from 25, I got the 403 error, but when I switched it back to 25, the results returned as expected. I think HomeHarvest's limit used to be 10k records and wonder whether HH or Realtor implemented some additional throttling to limit it from 10k to 1k. Are you able to return >1k records in one query without hitting the 403 @alphasynth ?

[alphasynth]

@dmr5090 - yes I am able to return >1k records without hitting the 403 error. Just ran a query for sold homes in San Antonio within the past 40 days which returned 1589 properties.

[dmr5090]

Ugh - OK, cool, thanks for confirming @alphasynth

[ZacharyHampton]

Try setting extra_property_data to False.