Internal buffer error memory allocation failed growing buffer

I’m working on a script which scrapes thousands of different web pages. Since these pages are usually different (has different sites), I use multithreading to speed up scraping.

EDIT: SIMPLE SHORT EXPLANATION

——-

I’m loading 300 urls (htmls) in one pool of 300 workers. Since the size of html is variable, sometimes, the sum of sizes is probably too big and python raises: internal buffer error : Memory allocation failed : growing buffer. I want to somehow check if this can happend and if, wait until the buffer is not full.

——-

This approach works but sometimes, python starts to throw:

internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer

into console. I suppose that it is because of size of html I store in memory, which can be 300*(for example 1mb) = 300mb

EDIT:

I know that I can decrease number of workers and I will. But it’s not a solution, there would be just lower chance to get such error. I want to avoid this error at all…

I started to log html sizes:

ram_logger.debug('SIZE: {}'.format(sys.getsizeof(html)))

And the result is (part):

2017-03-05 13:02:04,914 DEBUG SIZE: 243940
2017-03-05 13:02:05,023 DEBUG SIZE: 138384
2017-03-05 13:02:05,026 DEBUG SIZE: 1185964
2017-03-05 13:02:05,141 DEBUG SIZE: 1203715
2017-03-05 13:02:05,213 DEBUG SIZE: 291415
2017-03-05 13:02:05,213 DEBUG SIZE: 287030
2017-03-05 13:02:05,224 DEBUG SIZE: 1192165
2017-03-05 13:02:05,230 DEBUG SIZE: 1193751
2017-03-05 13:02:05,234 DEBUG SIZE: 359193
2017-03-05 13:02:05,247 DEBUG SIZE: 23703
2017-03-05 13:02:05,252 DEBUG SIZE: 24606
2017-03-05 13:02:05,275 DEBUG SIZE: 302388
2017-03-05 13:02:05,329 DEBUG SIZE: 334925

This is my simplified scraping approach:

def scrape_chunk(chunk):
    pool = Pool(300)
    results = pool.map(scrape_chunk_item, chunk)
    pool.close()
    pool.join()
    return results

def scrape_chunk_item(item):
    root_result = _load_root(item.get('url'))
    # parse using xpath and return

And the function to load html:

def _load_root(url):
    for i in xrange(settings.ENGINE_NUMBER_OF_CONNECTION_ATTEMPTS):
        try:
            headers = requests.utils.default_headers()
            headers['User-Agent'] = ua.chrome
            r = requests.get(url, timeout=(settings.ENGINE_SCRAPER_REQUEST_TIMEOUT + i, 10 + i), verify=False, )
            r.raise_for_status()
        except requests.Timeout as e:

            if i >= settings.ENGINE_NUMBER_OF_CONNECTION_ATTEMPTS - 1:
                tb = traceback.format_exc()
                return {'success': False, 'root': None, 'error': 'timeout', 'traceback': tb}
        except Exception:
            tb = traceback.format_exc()
            return {'success': False, 'root': None, 'error': 'unknown_error', 'traceback': tb}
        else:
            break

    r.encoding = 'utf-8'
    html = r.content
    ram_logger.debug('SIZE: {}'.format(sys.getsizeof(html)))
    try:
        root = etree.fromstring(html, etree.HTMLParser())
    except Exception:
        tb = traceback.format_exc()
        return {'success': False, 'root': None, 'error': 'root_error', 'traceback': tb}

    return {'success': True, 'root': root}

Do you know how to make it safe? Something which make workers wait if there would be buffer overflow problem?

Don’t reinvent the wheel and use Scrapy web-scraping framework:

Scrapy is an application framework for crawling web sites and
extracting structured data which can be used for a wide range of
useful applications, like data mining, information processing or
historical archival.

Think about it — the scalability/parallelizing/performance problem is solved for you — do you really want to continue diving into wonderful world of Threads in python, making your code smell suspiciously, hitting the CPU and memory limits, handling conflicts and, at the end, making your code impossible to debug and maintain — instead of focusing on the extracting and collecting the data? And, even with gevent I doubt your final code would be in any ways more simple and readable as the same you would implement based on Scrapy. Why not use the tool that was tested and used by a huge amount of users proven to be the best tool ever created in the web-scraping python world?

Here is a working spider that scrapes the quotes in a similar manner:

from scrapy.spider import Spider
from scrapy.item import Item, Field
from scrapy.http import Request


class BloombergItem(Item):
    ur = Field()
    parameter = Field()
    value = Field()


class BloombergSpider(Spider):
    name = 'bloomberg'
    allowed_domains = ['www.bloomberg.com']

    def start_requests(self):
        with open("stocks.txt") as f:
            for ur in f:
                yield Request("http://www.bloomberg.com/quote/%s:US" % ur)

    def parse(self, response):
        for parameter in response.css('table.key_stat_data tr'):
            item = BloombergItem()
            item['ur'] = response.xpath('//title/text()').extract()[0].split(':')[0]
            item['parameter'] = parameter.xpath('th/text()').extract()[0]
            item['value'] = parameter.xpath('td/text()').extract()[0]
            yield item

If the contents of stocks.txt is:

AAPL

The spider would output (for example, if you would choose to output it in JSON):

[
  {
    "parameter": "Current P/E Ratio (ttm)",
    "value": "16.6091",
    "ur": "AAPL"
  },
  {
    "parameter": "Estimated P/E(09/2015)",
    "value": "13.6668",
    "ur": "AAPL"
  },
  {
    "parameter": "Relative P/E vs.",
    "value": "0.9439",
    "ur": "AAPL"
  },
  ...
]

A good place to start with Scrapy is the Scrapy Tutorial.

Я работаю над скриптом, который сбрасывает тысячи различных веб-страниц. Поскольку эти страницы обычно разные (имеют разные сайты), я использую многопоточность, чтобы ускорить очистку.

EDIT: ПРОСТОЕ КОРОТКОЕ ПОЯСНЕНИЕ

——-

Я загружаю 300 URL-адресов (htmls) в одном пуле из 300 человек. Так как размер html является переменным, иногда сумма размеров, вероятно, слишком велика и повышает уровень python: internal buffer error: Memory allocation failed: growing buffer. Я хочу как-то проверить, может ли это произойти, и если, подождите, пока буфер не будет заполнен.

——-

Этот подход работает, но иногда питон начинает бросать:

internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer
internal buffer error : Memory allocation failed : growing buffer

в консоль. Я полагаю, что это из-за размера html я храню в памяти, который может быть 300 * (например, 1mb) = 300mb

РЕДАКТИРОВАТЬ:

Я знаю, что могу сократить число рабочих, и я это сделаю. Но это не решение, было бы меньше шансов получить такую ошибку. Я хочу избежать этой ошибки вообще…

Я начал регистрировать размеры html:

ram_logger.debug('SIZE: {}'.format(sys.getsizeof(html)))

И результат (часть):

2017-03-05 13:02:04,914 DEBUG SIZE: 243940
2017-03-05 13:02:05,023 DEBUG SIZE: 138384
2017-03-05 13:02:05,026 DEBUG SIZE: 1185964
2017-03-05 13:02:05,141 DEBUG SIZE: 1203715
2017-03-05 13:02:05,213 DEBUG SIZE: 291415
2017-03-05 13:02:05,213 DEBUG SIZE: 287030
2017-03-05 13:02:05,224 DEBUG SIZE: 1192165
2017-03-05 13:02:05,230 DEBUG SIZE: 1193751
2017-03-05 13:02:05,234 DEBUG SIZE: 359193
2017-03-05 13:02:05,247 DEBUG SIZE: 23703
2017-03-05 13:02:05,252 DEBUG SIZE: 24606
2017-03-05 13:02:05,275 DEBUG SIZE: 302388
2017-03-05 13:02:05,329 DEBUG SIZE: 334925

Это мой упрощенный метод скремблирования:

def scrape_chunk(chunk):
    pool = Pool(300)
    results = pool.map(scrape_chunk_item, chunk)
    pool.close()
    pool.join()
    return results

def scrape_chunk_item(item):
    root_result = _load_root(item.get('url'))
    # parse using xpath and return

И функция загрузки html:

def _load_root(url):
    for i in xrange(settings.ENGINE_NUMBER_OF_CONNECTION_ATTEMPTS):
        try:
            headers = requests.utils.default_headers()
            headers['User-Agent'] = ua.chrome
            r = requests.get(url, timeout=(settings.ENGINE_SCRAPER_REQUEST_TIMEOUT + i, 10 + i), verify=False, )
            r.raise_for_status()
        except requests.Timeout as e:

            if i >= settings.ENGINE_NUMBER_OF_CONNECTION_ATTEMPTS - 1:
                tb = traceback.format_exc()
                return {'success': False, 'root': None, 'error': 'timeout', 'traceback': tb}
        except Exception:
            tb = traceback.format_exc()
            return {'success': False, 'root': None, 'error': 'unknown_error', 'traceback': tb}
        else:
            break

    r.encoding = 'utf-8'
    html = r.content
    ram_logger.debug('SIZE: {}'.format(sys.getsizeof(html)))
    try:
        root = etree.fromstring(html, etree.HTMLParser())
    except Exception:
        tb = traceback.format_exc()
        return {'success': False, 'root': None, 'error': 'root_error', 'traceback': tb}

    return {'success': True, 'root': root}

Вы знаете, как сделать его безопасным? Что-то, что заставляет рабочих ждать, если проблема с переполнением буфера?

Description:
------------
Hi, i am receiving this message when i am trying to parse the 1.7gb xml file for inserting on aws server.
Memory allocation failed : growing buffer..
please help me out!

Thank You,

Sujal Jha

Test script:
---------------
<?php
include('../product/upload_function_backup.php');
include('../../config/database.php');
include('../../objects/hooks/function.php');
$dir = '../product_feeds/tata cliq';
$file= "tata_cliq_fashion.xml";
$data = file_get_contents($dir.'/'.$file);
$xml=stripslashes($data);
libxml_use_internal_errors(true);
$arr = simplexml_load_string((string)$xml);
foreach( libxml_get_errors() as $error ) {

    print_r($error);
}
print_r($arr);
?>


Expected result:
----------------
SimpleXMLElement Object
(
    [Product] => Array
        (
            [0] => SimpleXMLElement Object
                (
                    [ProductID] => 1065005367
                    [ProductSKU] => A08160
                    [ProductName] => WOMEN'S TRAINING STELLASPORT WOVEN SHORTS
                    [ProductDescription] => Stella McCartney inspired adidas SC Woven shorts for women. It comes with an Elasticated waist with two side pockets, one back zipped pocket and all-over 'Stellasport' branding.
                    [ProductPrice] => 1000.00
                    [ProductPriceCurrency] => INR
                    [WasPrice] => 2499.00
                    [DiscountedPrice] => 0.00
                    [ProductURL] => http://clk.omgt5.com/?AID=1327881&PID=17736&Type=12&r=https://shop.adidas.co.in/%23!product/A08160_scwovenshort
                    [PID] => 17736
                    [MID] => 929898
                    [ProductImageMediumURL] => https://content.adidas.co.in/static/Product-A08160/Women_Training_Shorts_A08160_1.jpg
                    [StockAvailability] => In stock
                    [Brand] => SPORT PERFORMANCE
                    [custom1] => Gender - Female
                    [custom2] => AgeGroup - Adult
                    [CategoryName] => SHORTS (1/4)
                    [CategoryPathAsString] => Root|TEXTILES|SHORTS (1/4)|
                )
)

Patches

add-fronk-support
(last revision 2018-11-28 13:15 UTC by sujaljhaonly4u at gmail dot com)

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports

this is a bugtracker and not a support forum!

contact your administrator and make sure you have *at least* twice the memory of stuff you want to proceed available and that there are no resource limits

Sorry, but your problem does not imply a bug in PHP itself.  For a
list of more appropriate places to ask for help using PHP, please
visit http://www.php.net/support.php as this bug system is not the
appropriate forum for asking support questions.  Due to the volume
of reports we can not explain in detail here why your report is not
a bug.  The support channels will be able to provide an explanation
for you.

Thank you for your interest in PHP.