Fetch data from db and third party api

Issue

I need to fetch all the owner ids and names, and their corresponding co-owners in txt file or json format. The owner ids and names are stored in the database but the corresponding co-owners can only be fetched by a third party api. And the third party apis take around 1 sec to fetch the data for a single owner id. I’m using Java 17 and spring boot; and Postgres as the database. I’ve around 10000 rows in the database.

For starters, I've used a loop to call the api and fetch data synchronously. But the api is timing out in this case as I have over 10k owner ids and for each owner id is taking around 1 sec. Then I’ve tried using multithreading in java but it’s still timing out.

Is there any other approach for these kind of problems statements?

Solution

A problem like this can be approached in a few ways:

1. Use multiple threads to make requests in parallel. This must be done carefully, because if you attempt to submit too many requests (i.e. from too many request threads) you can overload the server and increase response times ... or worse.

   You said that you tried multi-threading, but you gave no details, so we can only speculate as to why it didn't help, or whether you could address those (presumed) issues.

2. Change the API so that you request more information in each request. In your case, you might request information for multiple owner ids in one request. But this assumes that you can change the API to do that.

3. Get the information another way; e.g. by querying the 3rd-party system's database. This is probably impractical.

---

In your context, only approach 1 is likely to be feasible.

You could also see if you can reduce the average time taken to perform a single request. Possible things you could do include:

• Optimize / shorten the network path from your app to the 3rd-party server. (This assumes that network round-trip times or bandwidth limits are a bottleneck.)

• Use HTTP 1.0 keep-alives or HTTP 1.1 (or later) persistent connections. Starting a new connection for each call will slow down requests, not least because multiple client / server round trips are involved in establishing an SSL/TLS connection and authenticating.

Answered By - Stephen C