Draft idea for a use case of computational storage.


Sean Warnock
 

To All:

 

                This is my first submission to the group.  Attached is a rough draft for a use case and test scenario for computational storage.  I would welcome any feedback or ideas that the group has.

 

Sincerely,

Sean Warnock

 


thad.omura@...
 

Hi Sean, 

What is the Ethernet throughput to each Raspberry Pi?  I'm not too familiar with such a device but OSDs in Ceph typically require significant CPU and depending on use case, RAM.  I assume you are just looking at this for a POC?

http://docs.ceph.com/docs/jewel/start/hardware-recommendations 
- on CPU:  Ceph OSDs run the 
RADOS service, calculate data placement with CRUSH, replicate data, and maintain their own copy of the cluster map. Therefore, OSDs should have a reasonable amount of processing power (e.g., dual core processors).
- on Memory:  OSDs do not require as much RAM for regular operations (e.g., 500MB of RAM per daemon instance); however, during recovery they need significantly more RAM (e.g., ~1GB per 1TB of storage per daemon). Generally, more RAM is better.

To me (my opinion only) your use case is more about distributing computing (OSD) via Ethernet to smaller server instances (and likely smaller storage instances).  
I'd be interested to see if you make progress on distributing Ceph OSDs in this fashion. 

In terms of Computational Storage, our approach has been to look at workloads that specifically can gain performance/latency by having the compute right on the storage drive which is a little bit different.  That being said, would like to hear about your progress.

Regards,
Thad


Philip Kufeldt
 

At UC Santa Cruz's Center for Research in Open Source Software (CROSS) we are researching this exact question.  Under the Eusocial Project at CROSS, our first research project is creating a generic methodology for numerically quantifying the benefits of offloading tasks to smart storage devices.   This methodology can directly compare very disparate environments like general purpose compute systems with standard storage and systems that utilize smart storage devices.  The direct comparison is enabled by the creation canonical metrics based on standard work units that evaluate cost, power and space utilization.  The first experiments we are doing are specifically looking at offloading data management to a storage device, but the methodology can be used to evaluate any computational offload.  I have attached a poster we did for FAST19 and OCP Summit 19 describing our approach and showing some early results.  We expect a paper by this summer.


Another project at CROSS, SkyHook, is an example of a SQL DB that utilizes computational storage concepts.  SkyHook is a software layer between unmodified versions of Progres and Ceph.  This layer stores tables, creates distributed indexes and distributes queries.  It does this by sharding tables by row into objects. These objects are stored in Ceph thereby distributing the sharded table across many OSDs.  Now queries can be distributed to each of the OSDs via Ceph's standard API overload functionality.  This allows the query to operate on all of the table's objects in parallel on the OSDs.  In addition, indexing is done on each OSD, indexing the portions of the table they maintain. This further speeds up the distributed queries.

If you have and questions about this work at UCSC CROSS, let me know.


 

The work done at CROSS is very impressive, it was great to have a chance and participate at the even last year and learn about the efforts. The project for the ‘benchmarking’ was very impressive in the poster.

There are a lot of ways this technology shift can be used and benefit and efforts like this are a way to determine the best next steps.

That is why SNIA is involved to help lay an additional open source effort around ease of use and open doors to new ways not currently explored.

Exciting times.

 

Thank you,

Scott Shadley

VP Marketing

NGD Systems
M:    +1.208.570.3076 

CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient.

 

From: OCP-Storage@OCP-All.groups.io <OCP-Storage@OCP-All.groups.io> On Behalf Of pak via Groups.Io
Sent: Thursday, March 21, 2019 10:47
To: OCP-Storage@OCP-All.groups.io
Subject: Re: [OCP-Storage] Draft idea for a use case of computational storage.

 

At UC Santa Cruz's Center for Research in Open Source Software (CROSS) we are researching this exact question.  Under the Eusocial Project at CROSS, our first research project is creating a generic methodology for numerically quantifying the benefits of offloading tasks to smart storage devices.   This methodology can directly compare very disparate environments like general purpose compute systems with standard storage and systems that utilize smart storage devices.  The direct comparison is enabled by the creation canonical metrics based on standard work units that evaluate cost, power and space utilization.  The first experiments we are doing are specifically looking at offloading data management to a storage device, but the methodology can be used to evaluate any computational offload.  I have attached a poster we did for FAST19 and OCP Summit 19 describing our approach and showing some early results.  We expect a paper by this summer.


Another project at CROSS, SkyHook, is an example of a SQL DB that utilizes computational storage concepts.  SkyHook is a software layer between unmodified versions of Progres and Ceph.  This layer stores tables, creates distributed indexes and distributes queries.  It does this by sharding tables by row into objects. These objects are stored in Ceph thereby distributing the sharded table across many OSDs.  Now queries can be distributed to each of the OSDs via Ceph's standard API overload functionality.  This allows the query to operate on all of the table's objects in parallel on the OSDs.  In addition, indexing is done on each OSD, indexing the portions of the table they maintain. This further speeds up the distributed queries.

If you have and questions about this work at UCSC CROSS, let me know.


Sean Warnock
 

     I wanted to thank everyone that took a look at my rough idea.  After reading through CROSS’ projects there is already a large body of work in process.

 

Thad,

     My initial thought to use a Raspberry Pi for a Ceph OSD node is not a good choice.  I initially chose a Raspberry Pi for cost purposes.  After doing more research though, all bus interfaces on the Raspberry Pi are run through USB.  This significantly limits the 1Gb Ethernet interface performance. Second the only SATA bus options also run through the shared USB bus.  For these reasons I would not proceed with testing on a Raspberry Pi.

     While looking for another ARM based single board computer to test with, I found an existing product using an ARM single board computers for a Ceph cluster (see reference below).

Sean


On Thu, Mar 21, 2019 at 1:16 PM Scott Shadley <Scott.Shadley@...> wrote:

The work done at CROSS is very impressive, it was great to have a chance and participate at the even last year and learn about the efforts. The project for the ‘benchmarking’ was very impressive in the poster.

There are a lot of ways this technology shift can be used and benefit and efforts like this are a way to determine the best next steps.

That is why SNIA is involved to help lay an additional open source effort around ease of use and open doors to new ways not currently explored.

Exciting times.

 

Thank you,

Scott Shadley

VP Marketing

NGD Systems
M:    +1.208.570.3076 

CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient.

 

From: OCP-Storage@OCP-All.groups.io <OCP-Storage@OCP-All.groups.io> On Behalf Of pak via Groups.Io
Sent: Thursday, March 21, 2019 10:47
To: OCP-Storage@OCP-All.groups.io
Subject: Re: [OCP-Storage] Draft idea for a use case of computational storage.

 

At UC Santa Cruz's Center for Research in Open Source Software (CROSS) we are researching this exact question.  Under the Eusocial Project at CROSS, our first research project is creating a generic methodology for numerically quantifying the benefits of offloading tasks to smart storage devices.   This methodology can directly compare very disparate environments like general purpose compute systems with standard storage and systems that utilize smart storage devices.  The direct comparison is enabled by the creation canonical metrics based on standard work units that evaluate cost, power and space utilization.  The first experiments we are doing are specifically looking at offloading data management to a storage device, but the methodology can be used to evaluate any computational offload.  I have attached a poster we did for FAST19 and OCP Summit 19 describing our approach and showing some early results.  We expect a paper by this summer.


Another project at CROSS, SkyHook, is an example of a SQL DB that utilizes computational storage concepts.  SkyHook is a software layer between unmodified versions of Progres and Ceph.  This layer stores tables, creates distributed indexes and distributes queries.  It does this by sharding tables by row into objects. These objects are stored in Ceph thereby distributing the sharded table across many OSDs.  Now queries can be distributed to each of the OSDs via Ceph's standard API overload functionality.  This allows the query to operate on all of the table's objects in parallel on the OSDs.  In addition, indexing is done on each OSD, indexing the portions of the table they maintain. This further speeds up the distributed queries.

If you have and questions about this work at UCSC CROSS, let me know.



--
Sean