Web scraping and using APIs

Hi, will i get in trouble for scraping or using the apis that is found under network tab in chrome?
Is there any restriction written in terms and conditions or somewhere about this? i tried to quickly scan the condition and terms but itā€™s either i searched for the wrong term or it is not there. Planning to create a personal open-source app.

1 Like

They have previously encouraged the use for personal reasons.

I made a Screener the other year using the API, in this forum Iā€™ve posted ways to login with PHP and Guzzle.

@phildawson is correct, the apiā€™s are not intended forā€¦ ā€œbeing used like an apiā€ and there is not any official T&C for using it. It was asked/answered previously a couple of times that ā€œfor personal usageā€ it is ok.

Interesting, iā€™ll take both of your words for it then, until T212 says otherwise.

Also, @phildawson did you figure out how to get the tickerā€™s name with the api when you made your screener? I managed to get the tickers but i canā€™t seem to find how they resolve the ticker into their full name.

@kali by ā€œbeing used like an apiā€ i guess it means being used hundreds of times per minute?

1 Like

:thinking:

It used to be this:

https://live.trading212.com/rest/v2/instruments/search

1 Like

thatā€™s what I implied yes, there was this post of mine from ā€œa while agoā€ and there is the response like ā€œwe donā€™t want to do it half-arse, weā€™ll do it properly, hence we wonā€™t do it soon :)ā€

1 Like

Alright, thank you both of you @phildawson @kali for the clarifications.
My doubts are gone now.

No problems, although Iā€™ve just realised that /search endpoint doesnā€™t return everything now.

Iā€™ll just make a list of the endpoints I can see now.

Your reply helps a lot though, i searched for instruments and found it to be using https://live.trading212.com/rest/instruments/EQUITY/-1830575850 now, not sure what the numbers at the end means though.

Iā€™ve just quickly dumped what I can see from bundle-invest.js

array:146 [ā–¼
  0 => "auth/validate"
  1 => "v3/init-info"
  2 => "v3/account?includeMarkets=false"
  3 => "v3/account-registration/demo/associate/login"
  4 => "v3/account-registration/demo/associate/login"
  5 => "v3/account-registration/live/associate"
  6 => "v3/account/switch"
  7 => "v3/watchlists"
  8 => "v3/watchlists"
  9 => "v3/watchlists/{id}/metadata"
  10 => "v3/watchlists/{id}"
  11 => "v3/watchlists/{id}"
  12 => "v3/watchlists/order"
  13 => "v3/watchlists/{id}/visibility"
  14 => "v3/watchlists/{id}/tickers"
  15 => "v2/customer/isa/renew"
  16 => "v2/customer/documents/verify"
  17 => "v2/customer/documents/not-interested"
  18 => "v2/account-registration/details/countries"
  19 => "v2/account/save-profile"
  20 => "v2/account/isa/details"
  21 => "v2/account/instruments/settings"
  22 => "v2/account/change-password"
  23 => "v2/convert-coefficients"
  24 => "v2/instruments/price-increments"
  25 => "v2/instruments/additional-info/{ticker}"
  26 => "v2/trading/profit-losses"
  27 => "v2/deposit/generate-token"
  28 => "v2/payments/adyen/list-credit-cards"
  29 => "v2/languages"
  30 => "v2/time-zones"
  31 => "v2/push-notifications/registrations/web/{deviceId}"
  32 => "v2/push-notifications/registrations/{deviceId}"
  33 => "v2/push-notifications"
  34 => "v2/push-notifications/subscriptions"
  35 => "v2/chart-preferences/with-templates"
  36 => "v2/chart-preferences"
  37 => "v2/chart-preferences"
  38 => "v2/chart-preferences/"
  39 => "v2/chart-preferences/{preferencesId}"
  40 => "v2/chart-preferences/"
  41 => "v2/price-alerts/"
  42 => "v2/price-alerts/"
  43 => "v2/price-alerts/{alertId}"
  44 => "v2/price-alerts/{alertId}"
  45 => "v2/price-alerts"
  46 => "v2/notifications"
  47 => "v2/chart/templates/default/{templateName}"
  48 => "v2/chart/templates/default/reset"
  49 => "v2/chart/templates/reset"
  50 => "v2/chart/templates/{templateName}"
  51 => "v2/chart/templates/{templateName}"
  52 => "v2/two-factor-auth"
  53 => "v2/two-factor-auth"
  54 => "v2/two-factor-auth"
  55 => "v2/two-factor-auth"
  56 => "v2/two-factor-auth/backup?authenticationCode={authCode}"
  57 => "v2/customer-authentication/forget-all"
  58 => "v2/voucher-claims"
  59 => "v2/voucher-claims/{claimId}/accept"
  60 => "v2/voucher-claims/{claimId}/reject"
  61 => "v2/portfolio"
  62 => "v2/portfolio/info"
  63 => "v1/account/demo/type/pro"
  64 => "v1/account/demo/type/standard"
  65 => "v1/account/live/type/pro/apply"
  66 => "v1/account/share-lending"
  67 => "v1/account/pro-state"
  68 => "v1/account/settings"
  69 => "v1/account/reset-with-sum"
  70 => "v1/account/preferences"
  71 => "v1/addresses/main"
  72 => "v1/referrals/info"
  73 => "v1/trading-types"
  74 => "v1/nationality-data/validate"
  75 => "v1/nationality-data/collect"
  76 => "v1/logout"
  77 => "v1/share-lending-agreement"
  78 => "v1/share-lending-agreement/accept"
  79 => "v1/share-lending-agreement/reject"
  80 => "v1/brexit/consents"
  81 => "v1/brexit/consents/accept"
  82 => "v1/account-registration/appropriateness-test"
  83 => "v1/customer/registration-state"
  84 => "v1/customer/payment-state"
  85 => "v1/auto-w8/validate"
  86 => "v1/selfie-check/initialize"
  87 => "v1/selfie-check/complete"
  88 => "v1/pep-declaration"
  89 => "v1/equity/value-order/min-max"
  90 => "v1/equity/value-order/validate"
  91 => "v1/equity/value-order/review"
  92 => "v1/equity/value-order"
  93 => "v1/equity/value-order/{orderId}"
  94 => "v1/bank-accounts/gocardless"
  95 => "v1/recurring-payments/periods"
  96 => "v1/payments/gocardless/pending/sum"
  97 => "v1/batch/settings"
  98 => "v1/open-close-info/{workingScheduleId}"
  99 => "v1/exchanges/{accountType}/{version}"
  100 => "v1/notifications/{notificationIds}"
  101 => "v1/share-lending/allocations"
  102 => "v1/social/customer-settings"
  103 => "customer/accounts/funds"
  104 => "latest/customers/reviews/link"
  105 => "customers/deactivate"
  106 => "instruments/EQUITY/{version}"
  107 => "instruments/CFD-pro/{version}"
  108 => "instruments/CFD-retail/{version}"
  109 => "instruments/data/return-rates"
  110 => "deposit-declarations/state"
  111 => "companies/fundamentals"
  112 => "companies/full-ratios"
  113 => "companies/full-ratios"
  114 => "companies/full-financial/{statement}"
  115 => "companies"
  116 => "instrument-folders/EQUITY/{version}"
  117 => "instrument-folders/CFD/{version}"
  118 => "public/v2/equity/order/validate"
  119 => "public/v2/equity/order"
  120 => "public/v2/equity/order/{orderId}"
  121 => "public/v2/equity/order/{orderId}"
  122 => "public/added-costs"
  123 => "autoinvest/bucket-results/with-settings"
  124 => "autoinvest/buckets/limits"
  125 => "autoinvest/buckets/{bucketId}"
  126 => "autoinvest/buckets/{bucketId}/duplicate"
  127 => "autoinvest/buckets/{bucketId}/rebalance"
  128 => "autoinvest/buckets/{bucketId}/rebalance/{rebalanceId}"
  129 => "autoinvest/buckets/{bucketId}/actions/{actionId}"
  130 => "autoinvest/buckets/{bucketId}/limits/fund"
  131 => "autoinvest/buckets/{bucketId}/limits/draw"
  132 => "autoinvest/buckets/{bucketId}/fund"
  133 => "autoinvest/buckets/{bucketId}/draw"
  134 => "autoinvest/buckets/{bucketId}/fund"
  135 => "autoinvest/buckets/{bucketId}/draw"
  136 => "autoinvest/buckets"
  137 => "autoinvest/buckets/{bucketId}"
  138 => "autoinvest/buckets/{bucketId}/pause"
  139 => "autoinvest/buckets/{bucketId}/unpause"
  140 => "autoinvest/buckets/{bucketId}/recurring"
  141 => "autoinvest/buckets/{bucketId}/share"
  142 => "autoinvest/buckets/{bucketId}/share"
  143 => "autoinvest/buckets/{bucketId}/share"
  144 => "autoinvest/buckets/{bucketId}/import"
  145 => "autoinvest/buckets/{bucketId}/export"
]

Code if usefulā€¦

    	$file = file_get_contents(public_path('bundle-invest.js'));

    	preg_match_all('#"/rest/(.*?)"#', $file, $matches);

    	$grouped = [];

    	foreach ($matches[1] as $endpoint) {
    		
    		$array = explode('/', $endpoint);

    		$first = $array[0];
    		$second = $array[1] ?? '/';

    		$grouped[$first][$second][] = $endpoint;
    	}
2 Likes

Is has ā€œinstruments/EQUITY/{version}ā€, not sure what version refers to though.

@GreKomyu

Try with instrument-folders too :wink:

https://live.trading212.com/rest/instrument-folders/EQUITY/-1830575850

Holyā€¦ i donā€™t know how you did it (nvm, just saw the code, would still like 10 times) but if i could like 10 times i would.
Didnā€™t expect someone to just dump all of those here. This will help tremendously!
A really great help!

No worries.

I should add this looks at /rest so theres a few more.

There used to be these two, basically the same but you could query multiple to batch sending an array of instCode, periodType, limit inside the candles param.

/charting/rest/v2/candles
/charting/rest/batch

Looks like its now

/charting/v2/batch

I can see this too

/charting/prices

When Iā€™ve got some spare time I might create a new version.

Yea, i figured those two yesterday. Paired both of them with "deviations": [{"ticker": "", "useAskPrices": bool}] in the body of /batch and you can calculate the performance percentage for 1d, 1w, 1m, 3m, 6m, 1y time scale. Pretty neat.

I also figured out how the login api work, but i do have a couple of doubts around it. and i guess you donā€™t need it if you have the session tokenā€¦

If you have the session passed then you donā€™t need to login, and ridiculously it doesnā€™t expire.

Iā€™ve actually just figured out how different charts were from the original and its very close.

Other than changing the endpoint from /charting/rest/batch to /charting/v2/batch

New names for the params

instCode > ticker
periodType > period
limit > size
withFakes > includeFake

The timestamp used to be called birthTime in ms, now a normal timestamp and called ā€˜timestampā€™

Iā€™ll have to double check things but at first glance it seems to have kinda worked.

Need to hook up to the new /instruments/EQUITY/{version} next.

Something for another time.

3 Likes

That looks really good.
Mind if i messsage you in the future if i have more questions about the apis?

Sure, yeah Iā€™m happy to help if I can. I looked back in 2019/2020 so its changed a bit since.

I did think about making a bot to buy/sell automatically at the time.

I was interested in ProQuant but never used it on live funds before it closed.

You are doing Godā€™s work, my friend.

1 Like

To be honest, I have never heard before that there may be problems with the browser in this regard.