Web scraping and using APIs

[GreKomyu]

Hi, will i get in trouble for scraping or using the apis that is found under network tab in chrome?
Is there any restriction written in terms and conditions or somewhere about this? i tried to quickly scan the condition and terms but it’s either i searched for the wrong term or it is not there. Planning to create a personal open-source app.

[phildawson]

They have previously encouraged the use for personal reasons.

I made a Screener the other year using the API, in this forum I’ve posted ways to login with PHP and Guzzle.

Does trading212 have an API?

Easiest way is with Guzzle. [Screenshot 2020-11-20 at 14.40.04] use GuzzleHttp\Client; use GuzzleHttp\Cookie\CookieJar; private function postJSON($url, $payload) { $jar = CookieJar::fromArray( [ 'TRADING212_SESSION_LIVE' => $this->session ], 'trading212.com' ); $client = new Client(['cookies' => $jar]); $response = $client->post($url, ['json' => $payload,]); return json_decode($response->getBody(…

[kali]

@phildawson is correct, the api’s are not intended for… “being used like an api” and there is not any official T&C for using it. It was asked/answered previously a couple of times that “for personal usage” it is ok.

[GreKomyu]

Interesting, i’ll take both of your words for it then, until T212 says otherwise.

Also, @phildawson did you figure out how to get the ticker’s name with the api when you made your screener? I managed to get the tickers but i can’t seem to find how they resolve the ticker into their full name.

@kali by “being used like an api” i guess it means being used hundreds of times per minute?

[phildawson]

It used to be this:

https://live.trading212.com/rest/v2/instruments/search

[kali]

@GreKomyu

@kali by “being used like an api” i guess it means being used hundreds of times per minute?

that’s what I implied yes, there was this post of mine from “a while ago” and there is the response like “we don’t want to do it half-arse, we’ll do it properly, hence we won’t do it soon :)”

Please and very much please an API even a very basic one

I think it’d be a faily “easy-win” for the team since they already have the api’s. All that needs to be done is authenticate some api’s for users and rate limit them as liberally or conservatively as they wish.

[GreKomyu]

Alright, thank you both of you @phildawson @kali for the clarifications.
My doubts are gone now.

[phildawson]

No problems, although I’ve just realised that /search endpoint doesn’t return everything now.

I’ll just make a list of the endpoints I can see now.

[GreKomyu]

Your reply helps a lot though, i searched for instruments and found it to be using https://live.trading212.com/rest/instruments/EQUITY/-1830575850 now, not sure what the numbers at the end means though.

[phildawson]

I’ve just quickly dumped what I can see from bundle-invest.js

array:146 [▼
  0 => "auth/validate"
  1 => "v3/init-info"
  2 => "v3/account?includeMarkets=false"
  3 => "v3/account-registration/demo/associate/login"
  4 => "v3/account-registration/demo/associate/login"
  5 => "v3/account-registration/live/associate"
  6 => "v3/account/switch"
  7 => "v3/watchlists"
  8 => "v3/watchlists"
  9 => "v3/watchlists/{id}/metadata"
  10 => "v3/watchlists/{id}"
  11 => "v3/watchlists/{id}"
  12 => "v3/watchlists/order"
  13 => "v3/watchlists/{id}/visibility"
  14 => "v3/watchlists/{id}/tickers"
  15 => "v2/customer/isa/renew"
  16 => "v2/customer/documents/verify"
  17 => "v2/customer/documents/not-interested"
  18 => "v2/account-registration/details/countries"
  19 => "v2/account/save-profile"
  20 => "v2/account/isa/details"
  21 => "v2/account/instruments/settings"
  22 => "v2/account/change-password"
  23 => "v2/convert-coefficients"
  24 => "v2/instruments/price-increments"
  25 => "v2/instruments/additional-info/{ticker}"
  26 => "v2/trading/profit-losses"
  27 => "v2/deposit/generate-token"
  28 => "v2/payments/adyen/list-credit-cards"
  29 => "v2/languages"
  30 => "v2/time-zones"
  31 => "v2/push-notifications/registrations/web/{deviceId}"
  32 => "v2/push-notifications/registrations/{deviceId}"
  33 => "v2/push-notifications"
  34 => "v2/push-notifications/subscriptions"
  35 => "v2/chart-preferences/with-templates"
  36 => "v2/chart-preferences"
  37 => "v2/chart-preferences"
  38 => "v2/chart-preferences/"
  39 => "v2/chart-preferences/{preferencesId}"
  40 => "v2/chart-preferences/"
  41 => "v2/price-alerts/"
  42 => "v2/price-alerts/"
  43 => "v2/price-alerts/{alertId}"
  44 => "v2/price-alerts/{alertId}"
  45 => "v2/price-alerts"
  46 => "v2/notifications"
  47 => "v2/chart/templates/default/{templateName}"
  48 => "v2/chart/templates/default/reset"
  49 => "v2/chart/templates/reset"
  50 => "v2/chart/templates/{templateName}"
  51 => "v2/chart/templates/{templateName}"
  52 => "v2/two-factor-auth"
  53 => "v2/two-factor-auth"
  54 => "v2/two-factor-auth"
  55 => "v2/two-factor-auth"
  56 => "v2/two-factor-auth/backup?authenticationCode={authCode}"
  57 => "v2/customer-authentication/forget-all"
  58 => "v2/voucher-claims"
  59 => "v2/voucher-claims/{claimId}/accept"
  60 => "v2/voucher-claims/{claimId}/reject"
  61 => "v2/portfolio"
  62 => "v2/portfolio/info"
  63 => "v1/account/demo/type/pro"
  64 => "v1/account/demo/type/standard"
  65 => "v1/account/live/type/pro/apply"
  66 => "v1/account/share-lending"
  67 => "v1/account/pro-state"
  68 => "v1/account/settings"
  69 => "v1/account/reset-with-sum"
  70 => "v1/account/preferences"
  71 => "v1/addresses/main"
  72 => "v1/referrals/info"
  73 => "v1/trading-types"
  74 => "v1/nationality-data/validate"
  75 => "v1/nationality-data/collect"
  76 => "v1/logout"
  77 => "v1/share-lending-agreement"
  78 => "v1/share-lending-agreement/accept"
  79 => "v1/share-lending-agreement/reject"
  80 => "v1/brexit/consents"
  81 => "v1/brexit/consents/accept"
  82 => "v1/account-registration/appropriateness-test"
  83 => "v1/customer/registration-state"
  84 => "v1/customer/payment-state"
  85 => "v1/auto-w8/validate"
  86 => "v1/selfie-check/initialize"
  87 => "v1/selfie-check/complete"
  88 => "v1/pep-declaration"
  89 => "v1/equity/value-order/min-max"
  90 => "v1/equity/value-order/validate"
  91 => "v1/equity/value-order/review"
  92 => "v1/equity/value-order"
  93 => "v1/equity/value-order/{orderId}"
  94 => "v1/bank-accounts/gocardless"
  95 => "v1/recurring-payments/periods"
  96 => "v1/payments/gocardless/pending/sum"
  97 => "v1/batch/settings"
  98 => "v1/open-close-info/{workingScheduleId}"
  99 => "v1/exchanges/{accountType}/{version}"
  100 => "v1/notifications/{notificationIds}"
  101 => "v1/share-lending/allocations"
  102 => "v1/social/customer-settings"
  103 => "customer/accounts/funds"
  104 => "latest/customers/reviews/link"
  105 => "customers/deactivate"
  106 => "instruments/EQUITY/{version}"
  107 => "instruments/CFD-pro/{version}"
  108 => "instruments/CFD-retail/{version}"
  109 => "instruments/data/return-rates"
  110 => "deposit-declarations/state"
  111 => "companies/fundamentals"
  112 => "companies/full-ratios"
  113 => "companies/full-ratios"
  114 => "companies/full-financial/{statement}"
  115 => "companies"
  116 => "instrument-folders/EQUITY/{version}"
  117 => "instrument-folders/CFD/{version}"
  118 => "public/v2/equity/order/validate"
  119 => "public/v2/equity/order"
  120 => "public/v2/equity/order/{orderId}"
  121 => "public/v2/equity/order/{orderId}"
  122 => "public/added-costs"
  123 => "autoinvest/bucket-results/with-settings"
  124 => "autoinvest/buckets/limits"
  125 => "autoinvest/buckets/{bucketId}"
  126 => "autoinvest/buckets/{bucketId}/duplicate"
  127 => "autoinvest/buckets/{bucketId}/rebalance"
  128 => "autoinvest/buckets/{bucketId}/rebalance/{rebalanceId}"
  129 => "autoinvest/buckets/{bucketId}/actions/{actionId}"
  130 => "autoinvest/buckets/{bucketId}/limits/fund"
  131 => "autoinvest/buckets/{bucketId}/limits/draw"
  132 => "autoinvest/buckets/{bucketId}/fund"
  133 => "autoinvest/buckets/{bucketId}/draw"
  134 => "autoinvest/buckets/{bucketId}/fund"
  135 => "autoinvest/buckets/{bucketId}/draw"
  136 => "autoinvest/buckets"
  137 => "autoinvest/buckets/{bucketId}"
  138 => "autoinvest/buckets/{bucketId}/pause"
  139 => "autoinvest/buckets/{bucketId}/unpause"
  140 => "autoinvest/buckets/{bucketId}/recurring"
  141 => "autoinvest/buckets/{bucketId}/share"
  142 => "autoinvest/buckets/{bucketId}/share"
  143 => "autoinvest/buckets/{bucketId}/share"
  144 => "autoinvest/buckets/{bucketId}/import"
  145 => "autoinvest/buckets/{bucketId}/export"
]

Code if useful…

    	$file = file_get_contents(public_path('bundle-invest.js'));

    	preg_match_all('#"/rest/(.*?)"#', $file, $matches);

    	$grouped = [];

    	foreach ($matches[1] as $endpoint) {
    		
    		$array = explode('/', $endpoint);

    		$first = $array[0];
    		$second = $array[1] ?? '/';

    		$grouped[$first][$second][] = $endpoint;
    	}

[phildawson]

@GreKomyu

not sure what the numbers at the end means though.

Is has “instruments/EQUITY/{version}”, not sure what version refers to though.

[phildawson]

@GreKomyu

Try with instrument-folders too

https://live.trading212.com/rest/instrument-folders/EQUITY/-1830575850

[GreKomyu]

Holy… i don’t know how you did it (nvm, just saw the code, would still like 10 times) but if i could like 10 times i would.
Didn’t expect someone to just dump all of those here. This will help tremendously!
A really great help!

[phildawson]

No worries.

I should add this looks at /rest so theres a few more.

There used to be these two, basically the same but you could query multiple to batch sending an array of instCode, periodType, limit inside the candles param.

/charting/rest/v2/candles
/charting/rest/batch

Looks like its now

/charting/v2/batch

I can see this too

/charting/prices

When I’ve got some spare time I might create a new version.

[GreKomyu]

Yea, i figured those two yesterday. Paired both of them with "deviations": [{"ticker": "", "useAskPrices": bool}] in the body of /batch and you can calculate the performance percentage for 1d, 1w, 1m, 3m, 6m, 1y time scale. Pretty neat.

I also figured out how the login api work, but i do have a couple of doubts around it. and i guess you don’t need it if you have the session token…

[phildawson]

If you have the session passed then you don’t need to login, and ridiculously it doesn’t expire.

I’ve actually just figured out how different charts were from the original and its very close.

Other than changing the endpoint from /charting/rest/batch to /charting/v2/batch

New names for the params

instCode > ticker
periodType > period
limit > size
withFakes > includeFake

The timestamp used to be called birthTime in ms, now a normal timestamp and called ‘timestamp’

I’ll have to double check things but at first glance it seems to have kinda worked.

Need to hook up to the new /instruments/EQUITY/{version} next.

Something for another time.

[GreKomyu]

That looks really good.
Mind if i messsage you in the future if i have more questions about the apis?

[phildawson]

Sure, yeah I’m happy to help if I can. I looked back in 2019/2020 so its changed a bit since.

I did think about making a bot to buy/sell automatically at the time.

I was interested in ProQuant but never used it on live funds before it closed.

[andybrice]

You are doing God’s work, my friend.

[macebearfield0]

@GreKomyu

Hi, will i get in trouble for scraping or using the apis that is found under network tab in chrome?
Is there any restriction written in terms and conditions or somewhere about this? i tried to quickly scan the condition and terms but it’s either i searched for the wrong term or it is not there. Planning to create a personal open-source app.

To be honest, I have never heard before that there may be problems with the browser in this regard.