WP Content Crawler – Get content from almost any site, automatically
FOR WHAT IT CAN BE USED
- Create a personal site that collects news, posts, etc. from your favorite sites to see them in one place
- Use it with WooCommerce to collect products from shopping sites
- Collect products from affiliate programs to make money
- Collect posts to create a test environment for your plugin/theme
- Collect plugins, themes, apps, images from other sites to create a collection of them
- Keep track of competitors
- You can imagine anything. The internet is full of contents
HOW IT WORKS
It’s all about CSS selectors and you can learn how to use them in minutes by watching the introduction tutorial.
|Save every post detail|
Title, date, excerpt, content, tags, meta keywords, meta description, featured image, post images, custom meta… Just everything.
Just click on an element to find its CSS selector. You can also get alternative CSS selectors that you might be interested in. There is no need to leave your admin panel anymore.
Recrawl posts to keep them updated all the time. You can limit how many times a post can be updated, set update intervals, and ignore old posts.
|Custom post meta|
Save anything as a custom post meta. You can use a CSS selector or just type the value.
Use the artificial intelligence of Google Cloud Translation API or Microsoft Translator Text API to automatically translate the posts.
|Actions and filters|
If you are a developer, you can use actions and filters to extend and modify the plugin.
Prepare post content, title, excerpt, list item, and gallery item templates using shortcodes.
You can write alternative selectors to get the data even if the target site has post pages designed differently from each other.
|Find and replace anything|
You can use plain text or regular expressions to find and replace anything. You can even modify the HTML of the page, create your own HTML elements and write selectors to use them. You can even change image URLs. You have the power.
Does the target post have more than one page? No problem. You can save paginated posts as well.
|List type posts|
Some sites create posts with a list inside. You can extract the list from the post, create a template that should be applied to each list item and even reverse the list.
|Remove unnecessary elements|
Sometimes you need to get rid of some elements, such as advertisements, comments, you name it. Just write its CSS selector and it’s removed.
|Automatically insert category URLs|
Does the target site have hundreds of categories? No problem. Just write the CSS selector and the plugin will insert them for you.
Add unlimited sites and activate how many of them you want.
You can import and export site settings easily. Just copy and paste the code created by the plugin.
Set post type. It can be a post, a page, a product, or any other type.
You can remove links from the post. Just check the checkbox and the links are gone. That easy.
You can set a password for the posts to show them only to the users who have the password.
You can add notes for yourself to remind you things about the site. CSS selectors, TODO list, anything.
|Test everything on the fly|
Test post crawling, URL collection, CSS selectors, regular expressions, find and replace options, and proxies on the fly.
Using the tools, you can save posts manually with their URL, recrawl posts with their ID or delete already-saved URLs.
|Custom general settings for each site|
You can provide custom general settings for each post to override them and make them suitable for a site.
You can directly publish the saved posts or keep them as drafts to check them before publishing.
|Save images as a gallery|
You can save the images on the target page as a gallery and provide a template for each image to make it suitable for the gallery library that you use on the front end. You can also save the images as a WooCommerce gallery by just checking one checkbox.
|Duplicate post check|
Check duplicate posts by URL, post title, and/or post content.
You can add/remove minutes to/from the post date. In this way, you can schedule post publishing.
|Any data as a shortcode|
Get anything from the target page as a shortcode and use the shortcodes in templates to place any data anywhere you want.
Use a proxy or proxies to get content from the sites to which your IP does not have access. Currently supports HTTP and HTTPS proxies.
Attach cookies, such as session cookies, to each request.
|Crawl as many posts as you want|
You can set how many times post-crawling or URL collection CRON events should run. In this way, you can, e.g., save 100 posts every minute. Just be careful and consider your server’s capacity.
Set CSS selectors whose values should not be empty for category and post pages. When an empty value is found using those selectors, you can get an email notification.
See what’s going on in the background. Active sites, number of posts crawled, number of posts updated, last crawled and updated posts, last added URLs, last and next run of CRON events, currently being saved posts and URLs…
You can check the online documentation whenever you feel a need.
|Quick guides right next to the settings|
Each setting in the plugin has a quick guide that will help you understand what each setting does.
Watch video tutorials to easily learn how to use the plugin.
|Ready to translate|
You can translate the plugin into your own language using Poedit.
|Get updates from your admin panel|
You can update the plugin with just one click whenever an update is ready. Just go to your updates page in your admin panel.