From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.devel Subject: Re: persistent data feature Date: Sat, 11 Dec 2021 09:30:08 +0800 Message-ID: <87bl1olyhr.fsf@localhost> References: <87tufmjyai.fsf@gnus.org> <877dcil2sj.fsf@ericabrahamsen.net> <87czm98qi1.fsf@gnu.org> <87o85tcwm0.fsf@ericabrahamsen.net> <874k7ljwkr.fsf@gnus.org> <87fsr5cuzq.fsf@ericabrahamsen.net> <878rwx8mdn.fsf@gnu.org> <87r1aphuei.fsf@gnus.org> <837dcex6ub.fsf@gnu.org> <87bl1p10js.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="28293"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Richard Stallman , eric@ericabrahamsen.net, cesar.mena@gmail.com, emacs-devel@gnu.org, Pip Cet , Lars Ingebrigtsen , eliz@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Dec 11 02:29:47 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvrCt-0007B2-68 for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Dec 2021 02:29:47 +0100 Original-Received: from localhost ([::1]:57310 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mvrCs-0000nz-19 for ged-emacs-devel@m.gmane-mx.org; Fri, 10 Dec 2021 20:29:46 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:36150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mvrC0-00006A-Gy for emacs-devel@gnu.org; Fri, 10 Dec 2021 20:28:52 -0500 Original-Received: from [2a00:1450:4864:20::12d] (port=44810 helo=mail-lf1-x12d.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mvrBy-0000A8-SC; Fri, 10 Dec 2021 20:28:52 -0500 Original-Received: by mail-lf1-x12d.google.com with SMTP id z7so21011510lfi.11; Fri, 10 Dec 2021 17:28:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=hWl/UFTINAmtYzuMFBMbrCK3rsPTQzqPA91afICxVAM=; b=JsijHGmS4VmxFfpkg5t0YOJB2K8kjIvt+ru0bzGZy1UbdjNgCHwa5140Z59ePjGfyA Lm9pqYDEaM492utUBSV5TD4pWo0EUI3PZtXJl9QFgeIeXUM/N+ctZqBL9Qs1H9QH5+rj X5iznz1KpJ4XNCz4mUAd+bsEI+VcLveRr1pDwKqNr6zNpuqry9MqAQpGjuGF6vE3C1KC 5jXUB3D0Eygo/UmHNJ5l8Fyt6KdwDJ6RYfytqYfrsWzzROqLMbBXiY5uprXyFSiCMtgy US9Hs6ud9fepPT+gQ34kYej6ibtgCwKyCOU/BQ3DPErpgBNUCcVxHaoDJSJ7WiESVrxk Xo6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=hWl/UFTINAmtYzuMFBMbrCK3rsPTQzqPA91afICxVAM=; b=Fbazf2/S8ffCeNjHj3ZC+7TusboNGbF+P49manXXgHAMv3eSeWKO43nGjPqGRwuEKg bOjfeAg73gaYCZK4U+Bq3kQanwjc220KjtWxP+a/oVo+kYG9DWChRz+yZaZcYgtV+zuB sOyHa8ywRMKpyJ5MERjasaEVbTzCKyqHGiggyxa7jQsZ0KunaUWO1y3G8oTC00JSsIJk N0vIo+KQCcQd8UstH0QbhRrmN+oLLqKUtrKQD4RG0mvE7fg9oOgKuxvVkrq3TQDNvDFc JhneyRybM+43WwY3zPD/g2j6Ai7aiDMtE3lmwDGmts/UYzs6C7wQUjZFX+80wWnWUWNy NYkw== X-Gm-Message-State: AOAM533FLBHttR0Fmb4cOsl+U2ose7Kw/FN1ZoEW/y+rKMHik2n8v6NH H5N0GFG1LR1P9YmLtJBMKZI= X-Google-Smtp-Source: ABdhPJzdfRVy82U79e66oFS9jDasit3whiDZOpzwNl47Sc76/hD7c8gpk6PDitiXSPkWMRpxk6UgAw== X-Received: by 2002:a05:6512:2305:: with SMTP id o5mr16471205lfu.362.1639186125284; Fri, 10 Dec 2021 17:28:45 -0800 (PST) Original-Received: from localhost ([158.255.2.9]) by smtp.gmail.com with ESMTPSA id t7sm468633lji.43.2021.12.10.17.28.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 17:28:44 -0800 (PST) In-Reply-To: X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::12d (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::12d; envelope-from=yantar92@gmail.com; helo=mail-lf1-x12d.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:281634 Archived-At: Stefan Monnier writes: >> There will, of course, be large amounts of data, and that will be slow. >> (People already do this with a number things, and the results are >> predictably sluggish.) > > To help me understand this discussion, I think it would help me to have > examples of such large databases currently implemented as text files, > along with an idea of what "large" means in this context (how many MBs) > and where the "slow"ness manifests itself. I can think of two examples of large text databases: elfeed and recent org-mode's caching (org-persist). The slowness manifests itself on reading/printing(saving) the database and from the requirement to load the whole database into memory for queries. We are talking about 300Mb for elfeed and up to 20-30Mb per object (usually below 1Mb though) for org-persist with 100-200Mb total object size. Best, Ihor