outwit.com

The Macro Editor (pro version)

The macro editor allows you to create and edit macros.

In the top left corner of the Macro Editor Panel appears the name of the macro being edited, on a popup menu, which allows you to switch to any of the macros stored in your profile.

Start Page contains the URL of the first page to which your macro will be applied. If OutWit finds a possible next page link and if you have checked Browse or Slideshow in the Navigation Zone, then the macro will be applied to the whole series of pages.

Important Note: Start Page can also contain a Query Directory (in the form "queries/myDirectory") or a Query Generation Matrix. In these cases, the macro will be applied to all URLs of the directory or to all URLs generated by the matrix.

• Save Saves the current macro to your profile
• New Opens a new blank macro
• Reset Empties the macro editor
• Revert Restores the last saved version of the macro
• Delete Deletes the current macro from your profile
• Import Loads a previously exported macro
• Export Saves the macro as an XML file to your hard disk
• Get from Views Copies the current configuration of the bottom panels of all views to the extractors panel of the macro editor
• Send to Views Copies the current configuration of all extractors in the macro editor to the bottom panel of their respective views
• Properties Displays the Macro Properties Dialog where information about the current macro can be found and edited (name, author, comments, etc.)
• Manage Macros Displays the Automator Manager.

• Macro as URL (MAU)
• Navigation
• Catch
• Destination files
• Extractor settings

Macro as URL (MAU)

The Macro as URL is the representation of all the non-default settings as a single string that can be pasted in OutWit Hub's address bar for immediate execution, pasted in an email to be shared with other OutWit users etc. The MAU is updated in real time when the characteristics of the macro are changed in the macro editor. Reversely, editing the MAU will alter the definition of the macro in the user interface. An invalid syntax will cause the MAU to be displayed in red.

Important Note: You can execute a macro by simply pasting the MAU in the Hub's address bar and hitting return.

'Navigation' Zone

• Browse When checked, the macro will look for a next page link after the execution of all the active extractors in the current page, and so on, until no following page is found. The number of pages to explore this way can be limited in the Browse popup menu.
• Dig When checked, the active extractors will be applied to all links found in the start page (or of all browsed pages, if Browse is checked). The domain and depth of the links to explore can be defined in the Dig popup menu. The advanced settings dialog allows to set a criterion to visit only certain pages or to exclude certain pages from the automatic exploration.
• Slideshow When checked, the program will display a slideshow of the images found in the current page (or of all browsed pages, if Browse is checked).
• Fast Scraping In this mode exploration will be made without loading the full content of each pages, but only their textual part. This allows a much faster processing of large series of pages. When Fast Scraping is checked, the Browse, Dig and Slideshow checkboxes are disabled, as well as all extractors except for Scrapers.

  Note that browsing/digging and fast scraping are very different: applying a scraper by loading a page and going to the scraped view does the extraction from the source code of the page loaded, whereas using the 'Fast Scraping Mode' runs an XML HTTP Request for each URL, but doesn't really load the pages (ignoring images etc.). Most of the time, the result is the same, the Apply Scraper function being simply faster (hence the name). In some cases, however, the result can be different, or the Fast Scraping mode can even completely fail: the reason is that in the normal mode, events can happen that dynamically alter a page (mostly due to the execution of javascript scripts). These dynamic changes will not occur in the Fast Scraping mode as scripts are not executed. This means that dynamically added information, javascript redirections, page reloads... will simply not happen in Fast Scraping Mode. If you notice this kind of behavior, the best way is to accept the slower method and browse through the URLs, doing the scraping page after page.

'Catch Export' Zone

• Saving and Emptying the Catch before execution According to the option(s) checked, the content of the Catch can be emptied (and optionally saved to a backup file in the destination folder) before the execution of the macro. Note that it will not be restored after the execution, which means that you will have to load the backup file if you wish to recover the Catch content.
• Export When checked, the content of the Catch will be exported to a file after the execution of the macro. The destination file format can be defined in the Export popup menu. The export file will be saved in the destination file folder that you have selected in the Destination Files zone or in the current default folder if no destination folder is defined.

'Destination Files' Zone

This zone allows you to set the location and the name format for the files generated by your different extraction processes.

• Save extracted data to: If you set an export destination for one or several extractors in the macro or for the Catch, the generated files will be saved to the destination folder that you set here. For the case a file of the same name already exists, the popup menu will allow you to set the suffix to add to the filename.
• Split into several files: This checkbox and menu allow you to ask OutWit Hub to close the current export file and create a new one when the chosen condition becomes true. In the current version, the only criterion available is the number of ewtracted rows.
• Save downloaded files to: If you have instructed one or several extractors to download selected files, these will be saved in the folder you set here. For the case a file of the same name already exists, the popup menu will allow you to set the suffix to add to the filename.

'Extractor Setting' Panel

This panel contains the list of all extraction processes that can be performed during the execution of your macro.

• Extractor column: In this column, check the extractors that you wish to activate during the execution of the macro.
• Empty column: When checked (like in the views bottom panels), the data resulting from the extraction of one page will be cleared each time a new page is loaded.
• Options column: You can set, in this column, options that are specific to each extractor. These options correspond to the ones you will find in the bottom panel of each respective view in the Hub.
• Select If column: Allows you to set the criterion that will trigger the actions defined in the destination column (moving the selected data to the Catch, downloading files, exporting the data to a file). Like in the views bottom panels, you can set a selection criterion on either a specific field of the extractor or on all fields.
• Sorted by & Limit to columns: In association with the limit to option, the sort option allows you to set the sort column (field) and direction, in order to only extract the first n rows of extracted data.
• Destination column: Allows you to set the destination of the extracted data and files:
  • Catch: When checked, moves the selected data to the Catch (which can be exported to a file at the end of the process)
  • Download: When checked, tries to download all files corresponding to URLs found in the selected data to the set destination folder.
  • Export: Exports the selected data to a file in the Extracted data destination folder set above. You can set the export format using the Export popup menu.