Showing results for How to Check Website Rankings for Specific Pages
GitHub Repo
https://github.com/MalcmanIsHere/Python-Projct
MalcmanIsHere/Python-Projct
For people who had used this program before - the format of the config file has changed and you will need to re-download everything due to the new update mechanism. I apologize for the inconvenience. the program is multi-threaded; the default number of threads is your cpu cores * 3. You can temporarily change the number via the command line interface, or permanently change the number via the source code (in lib/deviantart.py at line 13) each artwork filename is appended with its artwork ID at the end for update validation purpose. The program downloads artworks for a user from newest to oldest until an existing file is found on the disk downloaded artworks are categorized by user and ranking mode modification time of each artwork is set according to upload order such that you can sort files by modified date ranking will overwrites existing files Instructions install Python 3.6+ install requests library pip install --user requests edit config.json file in data folder manually or via command line interface save directory: the save directory path users: the username shown on website or in URL Usage display help message $ python main.py -h usage: main.py [-h] [-f FILE] [-l] [-s SAVE_DIR] [-t THREADS] {artwork,ranking} ... positional arguments: {artwork,ranking} artwork download artworks from user IDs specified in "users" field ranking download top N ranking artworks based on given conditions optional arguments: -h, --help show this help message and exit -f FILE load file for this instance (default: data\config.json) -l list current settings -s SAVE_DIR set save directory path -t THREADS set number of threads for this instance display artwork help message $ python main.py artwork -h usage: main.py artwork [-h] [-a [ID ...]] [-d all [ID ...]] [-c all [ID ...]] optional arguments: -h, --help show this help message and exit -a [ID ...] add user IDs -d all [ID ...] delete user IDs and their directories -c all [ID ...] clear directories of user IDs display ranking help message usage: main.py ranking [-h] [-order ORDER] [-type TYPE] [-content CONTENT] [-category CATEGORY] [-n N] optional arguments: -h, --help show this help message and exit -order ORDER orders: {whats-hot, undiscovered, most-recent, popular-24-hours, popular-1-week, popular-1-month, popular-all-time} (default: popular-1-week) -type TYPE types: {visual-art, video, literature} (default: visual- art) -content CONTENT contents: {all, original-work, fan-art, resource, tutorial, da-related} (default: all) -category CATEGORY categories: {all, animation, artisan-crafts, tattoo-and- body-art, design, digital-art, traditional, photography, sculpture, street-art, mixed-media, poetry, prose, screenplays-and-scripts, characters-and-settings, action, adventure, abstract, comedy, drama, documentary, horror, science-fiction, stock-and-effects, fantasy, adoptables, events, memes, meta} (default: all) -n N get top N artworks (default: 30) download artworks from user IDs stored in config file; update users' artworks if directories already exist python main.py artwork download the top 30 (default) artworks that are popular-1-month, of type visual-art (default), of content original-work, and of category digital-art python main.py ranking -order popular-1-month -content original-work -category digital-art delete user IDs and their directories (IDs in users field + artwork directories), then download / update artworks for remaining IDs in config file python main.py artwork -d wlop trungbui42 add user IDs then download / update bookmark artworks for newly added IDs + IDs in config file python main.py artwork -a wlop trungbui42 use temp.json file in data folder as the config file (only for this instance), add user IDs to that file, then download / update artworks to directory specified in that file python main.py artwork -f data/temp.json -a wlop trungbui42 clear directories for all user IDs in config file, set threads to 24, then download artworks (i.e. re-download artworks) python main.py artwork -c all -t 24 Challenges there are two ways to download an image: (1) download button URL. (2) direct image URL. The former is preferred because it grabs the highest image quality and other file formats including gif, swf, abr, and zip. However, this has a small problem. The URL contains a token that turns invalid if certain actions are performed, such as refreshing the page, reopening the browser, and exceeding certain time limit Solution: use session to GET or POST all URLs for direct image URL, the image quality is much lower than the original upload (the resolution and size of the original upload can be found in the right sidebar). This is not the case few years ago when the original image was accessible through right click, but on 2017, Wix acquired DeviantArt, and has been migrating the images to their own image hosting system from the original DeviantArt system. They linked most of the direct images to a stripped-down version of the original images; hence the bad image quality. Below are the three different formats of direct image URLs I found: URL with /v1/fill inside: this means that the image went through Wix's encoding system and is modified to a specific size and quality. There are two cases for this format: old uploads: remove ?token= and the following values, add /intermediary in front of /f/ in the URL, and change the image settings right after /v1/fill/ to w_{width},h_{height},q_100. The width and height used to have a maximum limit of 5100 where (1) the system results in 400 Bad Request if exceeds the value, and (2) the original size will be returned if the required image is larger than the original. However, this has been changed recently. Now there is no input limit for the size, so you can request any dimension for the image, which may results in disproportional image if the given dimension is incorrect. In this case, I use the original resolution specified by the artist as the width and height new uploads: the width and height of the image cannot be changed, but the quality can still be improved by replacing (q_\d+,strp|strp) with q_100 Example: original URL vs incorrect dimension URL vs modified URL. The original url has a file size of 153 KB and 1024x1280 resolution, while the modified URL has a file size of 4.64 MB and 2700×3375 resolution. URL with /f/ but not /v1/fill: this is the original image, so just download it URL with https://img\d{2} or https://pre\d{2}: this means that the image went through DeviantArt's system and is modified to a specific size. I could not figure out how to get the original image from these types of links, i.e. find https://orig\d{2} from them, so I just download the image as is DeviantArt randomizes the div and class elements in HTML in an attempt to prevent scrapping, so parsing plain HTML will not work Solution: DeviantArt now uses XHR requests to send data between client and server, meaning one can simulate the requests to extract and parse data from the JSON response. The XHR requests and responses can be found in browsers' developer tools under Network tab. You can simply go to the request URL to see the response object age restriction Solution: I found that DeviantArt uses cookies to save the age check result. So, by setting the session.cookies to the appropriate value, there will be no age check sometimes the requests module will close the program with errors An existing connection was forcibly closed by the remote host or Max retries exceeded with url: (image url). I am not sure the exact cause, but it is most likely due to the high amount of requests sent from the same IP address in a short period of time; hence the server refuses the connection Solution: use HTTPAdapter and Retry to retry session.get in case of ConnectionError exception
GitHub Repo
https://github.com/Venu-Guptha/Indexing-Crawling-and-Ranking
Venu-Guptha/Indexing-Crawling-and-Ranking
Indexing is the process of adding web pages into Google search. Google tries to understand what the page is about and analyzes the content of the page, catalogs, images and video files implanted in a page. The Google Search index contains hundreds of billions of webpages and is well over 100,000,000 gigabytes in size. Search engines crawl the internet to discover keyword attached to websites and pages. These results are stored and organized into a database called an “index” for quick retrieval. Once content has been indexed, it can be served up on search engine results pages (SERPs) for relevant search queries. Today, Google Search can help you search text from millions of books from major libraries. In short, if you want your content to be found, it needs to be indexed for the opportunity to be seen. There are a few methods for inviting a search engine to crawl a page in order to be indexed more quickly. XML Site Maps Robot Meta Tags Fetch as Google Submit URL Hosting Content The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. Meta tags are essentially little content descriptors that help tell search engines what a web page is about. Meta elements are tags used in HTML and XHTML documents to provide structured metadata about a Web page. Crawling is the first part of having a search engine recognize your page and show it in search results. Crawling is the process by which Googlebot visits new and updated pages to be added to the Google index. New sites, changes to existing sites, and dead links are noted and used to update the Google index. Crawlers use algorithms to establish the frequency with which they scan a specific page and how many pages of the website it must scan. Googlebot is the web crawler software used by Google, which collects documents from the web to build a searchable index for the Google Search engine. Ranking - Once a keyword is entered into a search box, search engines will check for pages within their index that are a closest match; a score will be assigned to these pages based on an algorithm consisting of hundreds of different ranking signals. These pages (or images & videos) will then be displayed to the user in order of score. So in order for your site to rank well in search results pages, it’s important to make sure search engines can crawl and index your site correctly – otherwise they will be unable to appropriately rank your website’s content in search results.
GitHub Repo
https://github.com/vermogen01/How-Facebook-benefits-your-business-